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Preface 


This  report  documents  RAND’s  portfolio  analysis  tool  (PAT),  which  was  developed  at  the 
RAND  Corporation  for  the  Department  of  Defense  but  should  be  useful  in  other  strategic- 
planning  organizations  in  government  and  the  private  sector  as  well.  The  report  documents 
theory  and  methodology;  it  also  serves  as  a  combination  reference  manual  and  user  guide. 
In  a  sense,  it  is  a  second  edition,  because  PAT  builds  on  an  earlier  application-specific  tool, 
PAT-MD,  which  was  developed  for  the  U.S.  Missile  Defense  Agency’s  Program  Integration 
Office  (MDA/PI)  (Dreyer  and  Davis,  2005).  Since  PAT-MD  was  developed,  however,  we  have 
enhanced  it  substantially  and  have  used  it  in  a  number  of  very  different  projects  for  the  Office 
of  the  Secretary  of  Defense,  the  Joint  Staff,  the  U.S.  Air  Force,  and  the  Department  of  Home¬ 
land  Security.  Because  of  these  enhancements  and  expressions  of  interest  by  potential  users 
outside  of  RAND,  we  undertook  a  start-to-finish  revision  of  the  documentation. 

The  approach  to  analysis  enabled  by  PAT  is  oriented  toward  supporting  high-level  deci¬ 
sionmakers.  The  reasoning  behind  the  approach  is  described  in  this  report,  most  of  which, 
however,  is  technical  documentation  that  will  be  of  interest  primarily  to  analysts  and  those 
who  manage  analysis.  The  report  assumes  that  the  reader  is  at  least  moderately  familiar  with 
Microsoft  Excel . 

Since  PAT  is  an  evolving  tool,  questions  and  comments  are  especially  welcome.  They 
should  be  addressed  to  Paul  K.  Davis  (pdavis@rand.org)  or  to  the  developer,  Paul  Dreyer 
(dreyer@rand.org).  The  PAT  program  has  been  used  extensively  but  has  not  been  exhaustively 
tested.  The  documentation  was  written  using  Excel  2003  (Windows)  and  2004  (Macintosh);  a 
few  minor  differences  with  Excel  2007  (Windows)  are  mentioned  in  footnotes. 

This  research  was  sponsored  by  the  Office  of  the  Secretary  of  Defense  and  conducted 
within  the  Acquisition  and  Technology  Policy  Center  of  the  RAND  National  Defense  Research 
Institute,  a  federally  funded  research  and  development  center  sponsored  by  the  Office  of  the 
Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Department  of 
the  Navy,  the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence  Community. 

For  more  information  on  RAND’s  Acquisition  and  Technology  Policy  Center,  contact 
the  Director,  Philip  Anton.  He  can  be  reached  by  email  at  atpc-director@rand.org;  by  phone  at 
310-393-0411,  extension  7798;  or  by  mail  at  the  RAND  Corporation,  1776  Main  Street,  P.O. 
Box  2138,  Santa  Monica,  California  90407-2138.  More  information  about  RAND  is  available 
at  www.rand.org. 
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Summary 


Challenges  of  Strategic  Planning 

Strategic  planning  often  seeks  to  balance  investments  across  numerous  objectives.  Defense 
planners,  for  example,  have  objectives  relating  to  force  capabilities  for  future  traditional  and 
irregular  warfare  and  for  operations  other  than  war.  The  objectives  apply  separately  for  differ¬ 
ent  geographical  regions  and  time  periods.  Acquisition  planners  have  objectives  of  providing 
future  weapon-system  capabilities  in  each  of  many  mission  areas — again  for  different  opera¬ 
tional  circumstances  and  time  periods.  Trainers  have  objectives  such  as  preparing  troops  to 
operate  in  diverse  missions  and  circumstances.  None  of  these  planners  have  the  luxury  of  a 
single  objective  to  be  maximized.  Rather,  they  are  confronted  and  sometimes  confounded  by 
multiple  objectives,  few,  if  any,  of  which  can  be  ignored.  Nonetheless,  choices  must  be  made, 
because  resources  are  finite. 

Consequently,  strategic  planning  often  involves  investing  in  a  mix  of  capabilities  and 
activities  to  address  a  mix  of  objectives.  It  is  therefore  natural  to  use  the  terminology  of  port¬ 
folio  planning.  The  “portfolio”  itself  may  be  characterized  by  the  allocation  across  investment 
categories  (e.g.,  Army,  Navy,  Air  Force;  tanks,  ships,  and  planes)  or  by  the  corresponding  allo¬ 
cation  across  objectives  (e.g.,  traditional  versus  irregular  warfare).  In  either  case,  the  idea  is  to 
balance  the  portfolio.  This  does  not  mean  spreading  money  evenly  across  categories,  because 
not  all  objectives  are  equal  and  because  attending  adequately  to  one  may  require  much  less 
effort  than  doing  so  for  another.  Further,  given  a  large  baseline  of  investment  such  as  is  enjoyed 
by  the  Department  of  Defense  (DoD)  and  the  Department  of  Homeland  Security  (DHS), 
among  others,  some  ways  of  spending  a  marginal  billion  dollars  provide  far  more  leverage  than 
others.  Spending  (or  cutting)  that  marginal  billion  in  proportion  to  the  baseline  patterns  of 
expenditure  is  often  irrational.  Early  in  2009,  Secretary  of  Defense  Robert  Gates  made  this 
point  as  he  proposed  a  defense  budget  that  will  rebalance  the  portfolio  by  shifting  relatively 
small  resources  on  the  margin  toward  capabilities  for  irregular  warfare  and  stabilization,  secu¬ 
rity,  transition,  and  reconstruction. 


Dealing  with  Uncertainty  and  Disagreement 

Against  this  background,  portfolio  analysis  should  assist  decisionmakers  to  frame  their  think¬ 
ing  about  balance,  to  construct  good  multifaceted  options  for  consideration,  and  to  make 
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subsequent  choices.1  Such  analysis  must  include  solid  data  and  accurate  calculations,  but  it 
also  includes  subjective  inputs  and  analysis  under  deep  uncertainty.  Further,  it  must  deal  with 
the  reality  of  major  disagreements  among  senior  leaders,  disagreements  that  we  can  partially 
capture  under  the  rubric  of  strategic  perspective.  A  strategic  perspective  corresponds  analyti¬ 
cally  to  a  way  of  weighing  various  objectives  and  priorities  and  assessing  options’  adequacy  in 
meeting  them.  Much  of  this  is  about  managing  strategic  risk.  The  essence  of  strategic  decision¬ 
making  is  often  either  choosing  a  perspective  or  crafting  options  that  will  be  valuable  across 
important  perspectives.  Analysis  can  help  by  making  perspectives-related  issues  explicit  and,  in 
some  cases,  by  suggesting  nuanced  alternatives  that  are  seen  as  having  cross-perspective  value. 

RAND's  Portfolio  Analysis  Tool 

RAND’s  Portfolio  Analysis  Tool  (PAT)  was  designed  to  facilitate  strategic  portfolio  analysis 
dealing  with  both  uncertainty  and  differences  of  perspective.  It  reflects  lessons  learned  with 
earlier  tools  and  has  evolved  considerably  as  the  result  of  various  RAND  studies  since  2005. 

Comparing  Options  by  Various  Measures  and  by  Cost 

PAT  generates  high-level  summary  depictions  for  discussing  issues  of  balance.  It  uses  a 
spreadsheet-based  format  with  options  shown  in  rows  and  various  measures  of  option  good¬ 
ness  in  columns.  Figure  S.l  indicates  the  structure  of  PAT’s  Summary  sheet  schematically.  On 
the  left  is  a  five-color  scorecard;2  toward  the  right  are  optional  columns  for  specific  numerical 
outputs  of  interest  in  a  summary;  and  still  farther  to  the  right  are  column  groups  for  different 
depictions  of  cost  and  for  overall  effectiveness  across  measures  and  cost-effectiveness.  At  the 
top  left  are  various  “control  panels”  that  allow  the  user  to  change  the  way  underlying  calcula¬ 
tions  are  made,  as  well  as  a  number  of  other  items.  At  the  lower  left  is  a  color  bar  that  relates 
the  colors  to  underlying  effectiveness  scores  in  a  range  from  0  to  1. 

Drilling  Down 

PAT  makes  it  possible  to  drill  down  (zoom)  to  understand  the  basis  of  high-level  character¬ 
izations  and  to  put  a  spotlight  on  troublesome  problems.  The  mechanism  for  what  we  call 
the  drilldown  function  is  shown  schematically  in  Figure  S.2,  which  indicates  that  the  results 
shown  for  the  Measure  2  column  come  from  a  lower-level  (Level  2)  calculation  that  considers 
two  subordinate  measures,  M2.1  and  M2. 2.  Similarly,  the  results  shown  for  M2. 2  come  from 
an  underlying  calculation  that  considers  Level  3  measures.  A  user  looking  at  the  Summary 
sheet  can  click  on  a  column  to  bring  up  the  Level  2  scorecard  or  go  deeper  to  see  the  Level  3 
scorecard  of  interest  if  such  detail  is  provided. 

Such  drilldown  can  be  part  of  interactive  high-level  presentations  if  a  decisionmaker  chal¬ 
lenges  a  particular  assessment  out  of  interest  or  to  test  the  depth  of  staff  work.  Alternatively,  it 


1  The  primary  concepts  are  described  elsewhere  (Davis,  2002a;  Davis,  Kulick,  and  Egner,  2005;  and  Davis,  Shaver,  and 
Beck,  2008).  The  latter  applies  RAND’s  Portfolio  Analysis  Tool  (PAT)  to  acquisition  issues.  Another  publication  (Davis, 
Johnson,  Long,  and  Gompert,  2008)  uses  PAT  in  an  assessment  of  alternative  global  strategies.  Applications  are  ongoing 
with  the  U.S.  Air  Force  and  DHS. 

2  Red,  orange,  yellow,  light  green,  and  green  denote  very  poor,  poor,  marginal,  good,  and  very  good,  respectively.  The 
underlying  numerical  scores  may  also  be  shown,  but  they  are  usually  distracting  and  misleadingly  suggestive  of  precision. 
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Figure  S.1 

Schematic  View  of  PAT  Summary  Sheet 
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can  simply  be  part  of  staff  work  ensuring  that  analysis  is  clear,  well  grounded,  comprehensible, 
and  accompanied  by  an  audit  trail.  That  is,  undertaking  a  project  with  the  expectation  of  using 
PAT  can  help  structure  the  work  along  the  way. 

Alternative  Combining  Rules  (Aggregation  Rules) 

High-level  assessments  depend  on  more  detailed  assessments,  but  how  do  they  depend  on 
them?  When  information  is  sent  upward,  as  in  Figure  S.2,  mathematical  rules  must  be  speci¬ 
fied  for  doing  so.  Should  lower-level  assessments  simply  be  averaged?  Should  linear  weighted 
sums  be  used?  What  if  the  problem  itself  is  not  “linear”?  Options  for  systems  or  strategies  may 
have  zero  effectiveness  in  the  real  world  if  any  of  various  critical  components  fail.  A  better 
weapon  buys  nothing  if  it  cannot  be  delivered  to  the  target.  A  stabilization  strategy  may  fail  if 
any  of  its  political,  military,  or  economic  components  is  severely  lacking — despite  the  quality 
of  the  other  components.  The  natural  measures  in  PAT  analysis  often  include  some  that  are 
individually  critical  and  some  that  are  important  but  perhaps  substitutable.  Recognizing  such 
complexity,  we  designed  PAT  to  accommodate  a  wide  range  of  combining  rules.  Some  are  built 
in  and  can  be  chosen  from  a  menu;  others  may  be  analyst-defined. 

Alternative  Perspectives 

As  discussed  above,  the  concept  of  strategic  perspective  is  important  in  framing  issues  and 
evaluating  options.  In  our  work  with  PAT,  a  strategic  perspective  translates  into  a  coherent  set 
of  assumptions  about  the  way  objectives  are  weighed  and  assessments  of  capability  for  those 
objectives  calculated  and  combined  in  aggregating  from  lower  levels.  One  perspective,  for 
example,  might  emphasize  relatively  near-term  capabilities  and  activities  along  with  currently 
relevant  test  scenarios,  whereas  another  might  emphasize  longer-term  matters  and  do  so  using 
test  scenarios  with  intelligent  adversaries  responding  to  projected  U.S.  capabilities  and  vulner¬ 
abilities.  Ultimately,  perspectives  differ  in  the  value  placed  on  various  objectives,  judgments 
about  how  much  is  enough,  and  the  related  matter  of  managing  strategic  risk.  The  user  can 
also  define  “extended  perspectives”  that  make  distinctions  based  on,  e.g.,  how  the  effectiveness 
of  an  option  for  a  particular  measure  or  submeasure  is  estimated. 
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Figure  S.2 

Drilling  Down  for  Explanation 
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Multiresolution  Data  Entry  and  Modeling 

A  common  problem  in  strategic  decision  support  is  that  data — whether  empirical  and  objec¬ 
tive,  the  result  of  organizationally  approved  model  calculations,  or  the  result  of  the  structured 
judgment  of  experts — can  be  overwhelming.  Data  can  be  the  enemy  of  agile  analysis  intended 
for  sharp,  reductionist,  high-level  thinking.  PAT  is  designed  to  enable  data  to  be  entered  at 
alternative  levels  of  resolution.  Top-down  thinking  may  begin  with  rather  aggregated  assess¬ 
ments,  which  may  be  quite  sufficient  for  some  purposes.  Subsequently,  more  detailed  assess¬ 
ments  can  be  adopted,  in  which  case,  the  aggregated-level  results  stem  from  those  details. 
Alternatively,  a  bottom-up  approach  can  be  taken  initially,  but  more  aggregated  inputs  can  be 
used  for  quickly  addressing  “What  if .  .  .?”  questions.  In  practice,  greater  detail  is  only  some- 
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times  justified  and  can  even  be  misleading.  A  particular  option’s  technological  risk,  for  exam¬ 
ple,  may  best  be  judged  at  a  high  level,  because  that  risk  is  the  result  of  a  myriad  of  problematic 
issues,  not  all  of  which  are  even  identified. 


Illustrative  Outputs 

An  Illustrative  PAT  Summary  Sheet 

Figure  S.3  shows  an  illustrative  Summary  sheet  in  which  portions  that  may  ordinarily  be  sep¬ 
arated  horizontally  are  juxtaposed.  The  left  side  of  the  sheet  shows  that  options  are  being 
assessed  by  their  effectiveness  in  two  scenarios,  A  and  B,  and  by  something  called  “Other  Mea¬ 
sures,”  which  might  in  a  defense-planning  context  relate  to  something  like  shaping  the  inter¬ 
national  environment.  The  costs  shown  are  total  costs  for  a  20 -year  period.  The  effectiveness 
column  shows  an  “overall”  effectiveness  across  measures.  Relative  cost-effectiveness  compares 
the  cost-effectiveness  ratio  of  an  option  with  that  of  the  most  cost-effective  option  (except  for 
treating  the  baseline  as  0).  We  see  that  Option  2  is  by  far  the  most  cost-effective  (right  column) 
but  is  still  not  satisfactory,  as  indicated  by  the  yellow  and  red  cells  of  the  scorecard.  To  do  well, 
however  (Option  5),  is  quite  costly. 

As  noted  above,  the  values  asserted  for  the  aggregation  of  overall  effectiveness  depend 
not  only  on  the  assessments  for  each  individual  measure,  but  also  on  how  those  assessments 
are  combined.  Assumptions  can  be  changed  readily  in  either  sensitivity  analyses  or  more  com¬ 
prehensive  exploratory  analyses  in  which  all  key  variables  are  varied  simultaneously  to  explore 
what  has  been  variously  called  the  scenario  space,  outcome  space,  or  possibility  space.  If  such 
wide-ranging  exploration  has  been  accomplished  first  (whether  crudely  or  in  detail),  the  sce¬ 
narios  used  to  assess  the  options  can  be  carefully  defined  analytically  to  stress  the  options  in 
all  of  the  ways  regarded  as  important;  i.e.,  they  may  constitute  an  approximate  “spanning  set.” 
Current-day  official  planning  scenarios  do  not  typically  have  this  virtue,  but  they  could  have  it 
in  the  future.  The  number  of  test  scenarios,  of  course,  may  be  much  larger  than  the  number  of 
scenarios  in  the  illustrative  examples  in  this  report — at  least  at  the  working  level,  before  sim¬ 
plifications  are  introduced  for  communicating  efficiently  to  leadership. 

Cost-Effectiveness  Landscapes  as  a  Function  of  Perspective 

Displays  such  as  Figure  S.3  include  a  great  deal  of  information.  The  scorecard  information  is 
especially  important  because  decisionmakers  need  to  understand  how  the  options  fare  by  dif¬ 
ferent  criteria  (e.g.,  in  the  two  scenarios).  Balancing  across  such  measures  is  fundamentally  in 
the  province  of  decisionmakers,  and  they  are  ill-served  if  they  are  presented  only  with  super- 
aggregated  information  such  as  a  nonintuitive  single  metric  pretentiously  called  “effectiveness.” 
So  also,  the  drilldown  feature  is,  in  our  view,  essential. 

This  said,  after  decisionmakers  have  oriented  themselves  adequately  and  discovered  or 
asserted  their  preferences  and  the  alternative  strategic  perspectives  they  wish  to  take  seriously, 
it  can  be  quite  useful  to  generate  visuals  that  exploit  the  simpler  metric  of  overall  effectiveness. 
This  can  be  useful  for  communication  and  identifying  marginal  changes  to  improve  cost- 
effectiveness.  Figure  S.4  illustrates  this  with  what  we  call  a  “cost-effectiveness  landscape.”  It 
plots  the  overall  effectiveness  and  cost  of  each  option  and  does  so  for  each  of  two  strategic  per¬ 
spectives  (A  and  B).  If  enough  well-chosen  options  have  been  considered,  such  a  chart  can  com¬ 
municate  important  information  on  what  is  gained  as  a  function  of  expenditure.  The  results  for 
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Figure  S.3 

Illustrative  Composite  Summary  Sheet 
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Perspective  A  (upper,  dark-blue  diamonds)  indicate  that  Option  2  is  significantly  better 
than  Option  1  for  little  cost  and  that  incremental  improvements  occur  with  expenditures 
(Options  3  and  4).  Another  big  improvement  occurs  with  the  much  more  expensive  Option  5. 
In  contrast,  in  Perspective  B,  the  options  short  of  Option  5  have  much  less  value.  Perhaps  Per¬ 
spective  B  is  based  on  more  stressful  missions  and  more  conservative  assumptions.  Upon  look¬ 
ing  at  such  charts,  one  might  choose  to  talk  in  terms  of  minimum  and  “stretch”  goals,  shown 
by  the  horizontal  lines.  The  latter  would  be  feasible  only  if  a  larger  budget  were  forthcoming. 

Figure  S.4 

Cost-Effectiveness  Landscapes  for  Two  Perspectives 
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Risk  Management 

Risk  management  is  very  important  in  strategic  decisionmaking,  so  we  note  that  PAT  can 
depict  various  types  of  risk  in  a  number  of  ways,  e.g.,  with  explicit  top-level  measures,  with 
lower-level  measures  relating  to  more  stressful  versions  of  a  test  case,  by  showing  consequences 
of  more  and  less  conservative  combining  rules,  or  with  warning  flags  (as  shown  for  Option  5 
in  Figure  S.3). 

Well-conceived  work  with  PAT  should  convey  a  good  sense  of  what  is  being  assumed — a 
traditional  objective  in  analysis.  The  goal  in  decision  support,  however,  should  be  even  higher. 
It  is  one  thing  to  summarize  for  the  decisionmaker  the  assumptions  and  risks  associated  with 
an  option.  The  decisionmaker  is  better  served,  however,  if  he  or  she  is  presented  also  with 
options  that  mitigate  or  hedge  against  risks — even  risks  that  seem  unimportant  under  pre¬ 
vailing  best-estimate  reasoning.  Wise  decisionmaking  is  not  about  optimization  for  a  set  of 
assumptions;  it  is  about  finding  strategies  that  are  not  only  “likely”  to  be  acceptably  effective 
under  nominal  assumptions,  but  also  flexible,  adaptive,  and  robust. 
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CHAPTER  ONE 


Introduction 


Background 

Strategic  Planning  and  the  Balancing  of  Investments 

The  motivation  for  developing  RAND’s  Portfolio  Analysis  Tool  (PAT)  was  the  importance  of 
balancing  investments  across  multiple  objectives  in  strategic  planning.  This  balance  is  impor¬ 
tant  for  planning  in  such  distinct  domains  as  the  Department  of  Defense  (DoD),  the  Depart¬ 
ment  of  Homeland  Security  (DHS),  international  business  enterprises,  and  personal  finance. 
Today’s  defense  planners,  for  example,  have  objectives  relating  to  force  capabilities  for  future 
traditional  and  irregular  warfare  and  for  operations  other  than  war.  The  objectives  apply  sepa¬ 
rately  for  different  geographical  regions  and  time  periods.  Acquisition  planners  have  objectives 
of  providing  future  weapon-system  capabilities  in  each  of  many  mission  areas — again  for  dif¬ 
ferent  operational  circumstances  and  time  periods.  Trainers  have  objectives  such  as  preparing 
troops  to  operate  in  diverse  missions  and  circumstances.  None  of  these  have  the  luxury  of  a 
single  objective  to  be  maximized.  Rather,  planners  are  confronted  and  sometimes  confounded 
by  multiple  objectives,  few,  if  any,  of  which  can  be  ignored.  Nonetheless,  choices  must  be 
made,  because  resources  are  finite. 

Consequently,  strategic  planning  often  involves  investing  in  a  mix  of  capabilities  and 
activities  to  address  a  mix  of  objectives.  It  is  therefore  natural  to  use  the  terminology  of  port¬ 
folio  planning.  The  “portfolio”  itself  may  be  characterized  by  the  allocation  across  investment 
categories  (e.g.,  Army,  Navy,  Air  Force;  tanks,  ships,  and  planes)  or  by  the  corresponding  allo¬ 
cation  across  objectives  (e.g.,  traditional  versus  irregular  warfare).1  In  either  case,  the  idea  is  to 
balance  the  portfolio.  This  does  not  mean  spreading  money  evenly  across  categories,  because 
not  all  objectives  are  equal  and  because  attending  adequately  to  one  may  require  much  less 
effort  than  doing  so  for  another.  Further,  given  a  large  baseline  of  investment  such  as  that 
enjoyed  by  DoD  and  DHS,  among  others,  some  ways  of  spending  a  marginal  billion  dollars 
provide  far  more  leverage  than  others.  Spending  (or  cutting)  that  marginal  billion  in  propor¬ 
tion  to  the  baseline  patterns  of  expenditure  is  often  irrational.  Early  in  2009,  Secretary  of 
Defense  Robert  Gates  made  this  point  as  he  proposed  a  defense  budget  that  will  rebalance  the 


1  Soon  after  the  end  of  the  Cold  War,  it  was  useful  for  DoD  to  think  in  terms  of  balancing  across  the  objectives  of  shaping 
the  international  environment  (e.g.,  through  alliances  and  forward  deployment  of  U.S.  forces),  having  capabilities  appropri¬ 
ate  for  deterring  or  fighting  wars  in  the  near-  to  mid-term,  and  building  future  capabilities  to  deal  with  emerging  threats  and 
opportunities  (what  came  to  be  called  military  transformation).  An  early  analysis  proposing  such  portfolio  balancing  was 
Davis,  Gompert,  and  Kugler  (1996).  DoD’s  actual  strategy  was  called  Shape,  Respond,  and  Prepare  Now  (Cohen,  1997);  it 
specifically  addressed  all  three  objectives,  despite  the  erroneous  claim  by  outside  observers  that  it  was  merely  to  build  forces 
for  two  major  regional  conflicts  of  the  sort  considered  plausible  in  that  period. 
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portfolio  by  shifting  relatively  small  resources  on  the  margin  toward  capabilities  for  irregular 
warfare  and  stabilization,  security,  transition,  and  reconstruction. 

Dealing  with  Uncertainty  and  Disagreement 

Against  this  background,  portfolio  analysis  should  assist  decisionmakers  in  framing  their  think¬ 
ing  about  balance,  constructing  good  multifaceted  options  for  consideration,  and  making  sub¬ 
sequent  choices.2  Such  analysis  must  include  solid  data  and  accurate  calculations,  but  it  will 
also  include  subjective  inputs  and  analysis  under  deep  uncertainty.  Further,  it  must  deal  with 
the  reality  of  major  disagreements  among  senior  leaders,  disagreements  that  we  can  partially 
capture  under  the  rubric  of  strategic  perspective.  A  strategic  perspective  corresponds  analyti¬ 
cally  to  a  way  of  weighing  various  objectives  and  priorities  and  assessing  options’  adequacy 
in  meeting  them.  Much  of  this  perspective  is  about  managing  strategic  risk.  The  essence  of 
strategic  decisionmaking  is  often  either  choosing  a  perspective  or  crafting  options  that  will  be 
valuable  across  important  perspectives.  Analysis  can  help  by  making  perspectives-related  issues 
explicit  and,  in  some  cases,  by  suggesting  nuanced  alternatives  that  are  seen  as  having  cross¬ 
perspective  value. 


Portfolio  Analysis  Tools 

Definitions 

We  use  the  term  “portfolio  analysis  tool”  to  mean  a  tool  for  comparing  investment  options 
according  to  a  number  of  quantitative  and  qualitative  criteria,  including  costs,  upside  poten¬ 
tial,  and  downside  potential  (risk).  In  strategic  planning,  such  a  tool  can  generate  holistic,  top- 
down  depictions  of  alternatives  and  their  possible  implications,  perhaps  over  many  years  into 
the  future.  Such  a  tool  can  assist  in  balancing  programs,  either  with  a  start-fresh  approach  or 
in  marginal  analysis,  i.e.,  assessing  where  to  add  or  subtract  the  marginal  dollar. 

RAND’s  approach  to  strategic  portfolio  analysis  has  evolved  over  the  past  dozen  years.  It 
has  been  applied  at  several  very  different  levels  of  analysis: 

•  Force  planning  for  the  Office  of  the  Secretary  of  Defense  in  the  mid-1990s  (Davis, 
Gompert,  and  Kugler,  1996;  Davis,  Kugler,  and  Hillestad,  1997) 

•  Strategic  planning  for  the  Missile  Defense  Agency  (MDA) 

•  Acquisition-level  planning  for  Prompt  Global  Strike  (Davis,  Shaver,  and  Beck,  2008) 

•  Resource-informed  strategic  assessments  for  the  Joint  Staff  (Davis,  Johnson,  Long,  and 
Gompert,  2008) 

•  Mission-level  analysis  for  the  U.S.  Air  Force  and  DHS  (Davis,  Hillestad,  Long,  Dreyer, 
and  Dues,  forthcoming). 

Our  mid-1990s  work  was  based  on  DynaRank,  a  decision  support  system  developed 
primarily  by  Richard  Hillestad  at  RAND  (Hillestad  and  Davis,  1998;  Miller,  2007).  A  par- 


2  The  primary  references  for  this  report’s  concepts  regarding  strategic-level  decision  support  are  Davis  (2002a),  Davis, 
Kulick,  and  Egner  (2005),  and  Davis,  Shaver,  and  Beck  (2008).  The  latter  includes  an  application  of  PAT  to  acquisition 
issues.  Another  recent  publication — Davis,  Johnson,  Long,  and  Gompert,  2008 — includes  an  application  to  assessment  of 
alternative  global  strategies. 
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allel  stream  of  RAND  research  at  the  time  used  the  objectives-to-programs  methodology.3 
An  improved  tool,  PAT-MD,  was  developed  for  MDA  (Dreyer  and  Davis,  2005),  along  with 
an  integrated-capabilities  model  for  missile  defense,  CAM-MD  (Willis,  Bonomo,  Davis,  and 
Hillestad,  2006).  RAND’s  work  for  MDA  motivated  the  Under  Secretary  of  Defense  for  Acqui¬ 
sition,  Technology,  and  Logistics  to  request  development  of  a  generic  version  of  the  approach 
and  tool  for  capabilities  analysis.  That  became  PAT,  the  subject  of  this  report. 

Functionality  of  PAT 

PAT  is  not  a  model  in  the  usual  sense;  rather,  it  is  a  cross-platform  spreadsheet  tool  (built  in 
Microsoft  Excel)4  that  facilitates  planning  by  presenting  information  in  a  way  that  is  useful  to 
senior  leaders.  However,  using  PAT  encourages  a  structured  way  of  thinking  that  generates  a 
conceptual  model  for  the  problem  being  analyzed.  Further,  PAT  can  use  a  variety  of  separate  or 
embedded  models  as  sources  of  input  data.  PAT  is  an  empty  vessel,  but  one  with  many  useful 
features: 

1.  Summary  scorecards.  PAT  generates  stoplight  charts,  simple  color-scorecard  summa¬ 
ries  of  how  options  rate  on  a  number  of  juxtaposed  criteria,  such  as  measures  of  capabili¬ 
ties,  risks,  upside  potential,  and  costs.  These  criteria  may  be  quantitative  or  qualitative, 
objective  or  subjective. 

2.  Drilldown  (zooming).  PAT  generates  its  summaries  from  more  detailed  consider¬ 
ations,  which  can  be  viewed  by  drilling  down  to  a  level  that  provides  assumptions,  a 
terse  logic,  and  a  measure  of  rigor,  even  for  qualitative  assessments.  Two  levels  of  drill¬ 
down  are  available. 

3.  Multiresolution  modeling  (MRM)  and  data  entry.  PAT  allows  the  analyst  to  enter 
data  at  a  lowest  level  of  detail  or  at  one  of  two  more-aggregated  levels.  Entering  data  at 
the  more-aggregated  levels  reduces  the  amount  of  data  entry  greatly  and  is  consistent 
with  the  time-honored  analytic  approach  of  working  top-down  (starting  at  a  high  level, 
then  adding  enriching  detail  where  warranted).  The  data  themselves  may  be  generated 
by  a  multiresolution  model  or  family  of  models. 

4.  Sensitivity  analysis  and  exploratory  analysis.  PAT  allows  the  analyst  to  quickly 
recognize  key  assumptions  and  to  change  them  interactively.  This  may  be  done 
parameter-by-parameter  or  more  broadly.  These  analyses  are  greatly  facilitated  by  the 
MRM  feature. 

5.  Alternative  aggregation  methods.  PAT  allows  the  analyst  to  quickly  change  how 
summary  depictions  are  generated  (i.e.,  how  they  are  aggregated  from  details).  Choices 
include,  for  example,  simple  linear  weighted  sums,  some  nonlinear  “weakest  link”  meth¬ 
ods,  linear  weighted  sums  with  threshold  constraints,  and  rank  ordering.  The  analyst 
can  also  use  customized  aggregation  rules;  i.e.,  PAT  is  extensible.  We  have  found  that 
to  be  important  in  practice. 


3  A  mid-1990s  RAND  tool,  the  objectives-to-programs  methodology  used  utility  functions  and  spreadsheet  methods  to 
arrive  at  a  higher  level  of  aggregation  (unpublished  work  by  Manuel  Carrillo  and  Preston  Niblack,  1996). 

1  This  documentation  is  intended  for  Excel  2003  (Windows)  and  Excel  2007  (Macintosh),  except  for  a  few  differences  in 
Excel  2007  (Windows)  (noted  in  footnotes),  with  which  we  have  had  relatively  little  experience.  PAT  will  not  work  with 
Excel  2008  (Macintosh)  because  it  does  not  support  Visual  Basic  macros.  However,  Macintosh  users  can  use  PAT  readily 
within  a  virtual  machine,  such  as  Parallels  Desktop™  or  VMWare  Fusion™. 
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6.  Links  to  capability  analysis  and  other  sources  of  data.  PAT  links  to  even  more- 
detailed  information,  such  as  that  of  an  embedded  or  connected  capabilities  model,  data 
generated  separately  from  a  capabilities  model,  empirical  data,  or  structured  judgments.5 

7.  Marginal  analysis.  Although  PAT  emphasizes  multiobjective  scorecards,  it  also  gener¬ 
ates  scores  of  overall  effectiveness  or  cost-effectiveness.  These  can  be  used  for  marginal 
or  chunky  marginal  analysis  about  how  to  spend  (or  cut)  the  next  increment  of  funds. 

8.  Ability  to  represent  and  contrast  alternative  perspectives.  PAT  encourages  analysts 
to  deal  explicitly  with  the  serious  differences  of  opinion  and  judgment  that  can  be 
referred  to  as  alternative  perspectives.  Results  of  PAT  analysis  can  then  be  shown  as  a 
function  of  strategic  perspective.  There  can  be  striking  differences  in  implications  for 
cost-effectiveness  assessments. 

9.  Facilitated  operations.  At  a  mechanical  level,  PAT  automates  many  tedious  spread¬ 
sheet  operations  so  that  users  can  generate  and  manipulate  portfolio-style  scorecards  and 
underlying  detailed  information  quickly.  It  also  provides  a  variety  of  built-in  displays. 


Report  Structure 

The  report  is  organized  as  follows.  Chapter  Two  introduces  the  principal  concepts  and  terms 
in  PAT  and  gives  a  schematic  overview.  Chapters  Three  and  Four  then  describe  PAT’s  input 
and  outputs  in  user-manual  detail.  Chapter  Five  discusses  selected  theory  and  methods  in 
more  detail — especially  aggregation  methods  and  methods  for  marginal  and  chunky  marginal 
analysis.  Chapter  Six  wraps  up  with  suggestions  and  cautions  for  users  and  with  thoughts 
about  future  work.  Appendix  A  is  a  QuickStart  exercise  for  those  who  like  to  learn  by  doing. 
Appendixes  B  and  C  describe  some  practical  hints  for  those  who  are  actually  using  PAT. 


5  An  embedded  model  might  be  implemented  in  a  particular  worksheet  of  the  PAT  workbook.  A  “connected”  model  might 
import  data  from  a  program  written  in  a  different  language,  such  as  Analydca®  (a  product  of  Lumina  Decision  Systems,  Inc., 
[www.lumina.com]).  We  used  such  a  connection  approach  in  our  work  for  MDA  (Willis  et  ah,  2006).  A  capability  model 
built  in  Analytica  (CAM-MD)  could  be  exported  to  PAT-MD. 


CHAPTER  TWO 


Overview  of  PAT 


Inputs  and  Outputs 

PAT  takes  a  series  of  inputs  and  generates  outputs  in  the  form  of  portfolio-style  tables  and  vari¬ 
ous  charts  and  graphics  (Figure  2.1).  That  is,  viewed  as  a  “black  box,”  it  primarily  generates 
displays  to  describe  implications  of  input  information  in  a  structured  way. 

Many  of  the  inputs,  such  as  the  investment  options  to  be  compared,  are  what  one  might 
expect.  A  given  investment  option  specifies  expenditures  in  each  budget  category  for  each  year 
covered  by  the  analysis.  This  could  include,  e.g.,  separate  expenditures  in  the  budget  categories 
of  research  and  development  (R&D),  acquisition,  and  operations  and  support  (O&S).  Invest¬ 
ment  options  may  differ  in  what  is  to  be  developed  and  how  fast,  in  what  will  be  deployed 
operationally,  and  so  on.  Or  they  may  differ  because  of  alternative  technical  approaches  or 
because  of  alternative  strategies. 

As  shown  in  Figure  2.1,  other  inputs  to  PAT  include  capabilities,  risks,  and  costs  for  each 
investment  option,  as  well  as  “control  parameters,”  which  determine  the  form  of  the  outputs, 
the  assumptions  and  methods  used  for  evaluation  and  aggregation,  and  so  forth.  They  can 
strongly  affect  how  the  various  options  stack  up  in  summary  displays. 

As  indicated  schematically  in  Figure  2.2,  PAT’s  outputs  include  color-coded  scoreboards, 
which  compare  options  by  different  objectives  or  measures  (Columns  A,  B,  and  C),  with  red 
indicating  poor  and  green  indicating  good;  tabular  outputs  on  overall  effectiveness  and  cost; 
and  standard  charts,  such  as  charts  of  cost  versus  time.  Many  more  types  of  output  display  are 
available  or  can  be  readily  constructed;  we  describe  those  more  fully  in  subsequent  chapters. 

Figure  2.1 

PAT  as  a  Black  Box 

Investment  options 
Capability  assessments 


Upside  potential 

Risk  assessments 
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Cost  data 

Graphic  displays 

Control  paramaters  for 

Effectiveness  comparisons 

aggregations,  formats... 

PAT 

Effectiveness  versus  cost  comparisons 
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Figure  2.2 

Illustrative  Output  Types 
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We  next  introduce  a  number  of  basic  concepts  and  terms.  Chapter  Five  provides  a  more 
complete  and  rigorous  treatment. 


Concepts  and  Terminology 

Multicriteria  Scorecards 

Strategic  planning  typically  requires  evaluating  options  by  multiple  criteria.  These  criteria 
may  relate  to  different  objectives,  operations,  circumstances,  time  scale,  and  so  on.  They  may 
include  measures  of  risk  and  upside  potential.1  Although  classic  cost-effectiveness  methods 
emphasize  combining  the  effectiveness  scores  for  different  criteria  to  obtain  a  single  variable 
to  be  optimized,  modern  policy  analysis  has  long  emphasized  “policy  scorecards,”  because 
decisionmakers  need  to  see  how  the  options  fare  by  the  different  criteria.2  The  relative  good¬ 
ness  of  options  may  eventually  be  summarized  in  terms  of  a  single  index  or  utility,  but  that 
simplification  should  follow  more  discriminate  reasoning  with  multiple  criteria.  The  reason  for 
this  sequence  is  that  the  cross-cutting  thinking  across  criteria  is  precisely  what  strategic  deci¬ 
sionmakers  are  often  most  concerned  about  and  most  uniquely  responsible  for.  Such  thinking 
is  not  mere  mathematical  problem-solving. 


1  Both  risk  (essentially  downside  potential)  and  upside  potential  should  be  considered  in  good  decisionmaking  (Davis, 
Kulick,  and  Egner,  2005),  but  it  is  common  (even  in  schoolhouses)  for  one  or  the  other  to  be  given  short  shrift,  which  biases 
the  analysis. 

2  The  late  Bruce  Goeller  helped  pioneer  work  on  such  policy  scorecards  in  the  1970s  (Goeller  et  ah,  1983;  Dunn  and  Kelly, 
1991,  pp.  133  ff). 
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It  is  cognitively  efficient  to  use  policy  scorecards  that  are  color-coded  in  the  familiar  way 
(red  means  bad,  yellow  means  marginal,  and  green  means  good),  although  PAT  has  options  for 
generating  scorecards  in  other  formats.  Interestingly,  such  scorecards  are  sometimes  criticized, 
but  for  reasons  that  do  not  apply  to  our  usage.  The  primary  problem  is  that  officials  are  often 
briefed  with  scorecard-based  analysis  that  is  “one  viewgraph  deep,”  with  little  if  any  discussion 
of  what  (if  anything)  underlies  the  color-coding  of  results.  Our  approach  is  different. 

Multiresolution  Thinking 

Four  Levels  of  Detail.  Each  investment  option  can  be  evaluated  in  PAT  at  up  to  four  levels  of 
detail:  (overall)  effectiveness,  Level  1,  Level  2,  and  Level  3.  Ligure  2.3  illustrates  this  schemati¬ 
cally  with  an  example  that  we  use  throughout  this  report.  Results  at  a  given  level  are  either 
specified  directly  or  calculated  from  the  next  lower  level  (e.g.,  results  from  Level  3  “roll  up” 
or  aggregate  to  results  at  Level  2,  and  those  from  Level  2  aggregate  to  results  at  Level  1).  This 
allows  drilldown  (zooming).  That  is,  a  result  at  one  level  can  be  explained  by  drilling  down  to 
deeper  levels. 

The  focus  of  our  work  is  at  Level  1,  Level  2,  and  Level  3.  At  these  levels,  PAT’s  displays 
assess  the  options  by  a  number  of  criteria  called  “measures.”  At  Level  1 ,  for  example,  an  illustra¬ 
tive  measure  might  be  effectiveness  in  Scenario  A;  another  might  be  risks  (i.e.,  downside  poten¬ 
tial,  the  potential  for  doing  worse  than  is  shown  in  the  effectiveness  columns).  More  extensive 
examples  are  presented  later. 

Multiresolution  Data  Entry.  The  straightforward  way  to  build  such  a  multilevel  system 
of  outputs  is  bottom-up,  which  typically  means  developing  detailed  data  sheets.  Those,  how¬ 
ever,  can  be  tedious  to  build  and  to  use,  especially  if  uncertainty  analysis  is  needed.  Lurther, 
the  complexity  and  detail  of  such  sheets  can  result  in  errors  as  analysts  vary  assumptions.  We 
designed  PAT  to  have  multiresolution  data  entry.  This  means  that  PAT  maintains  several  data 
sheets.  At  any  given  time,  PAT  will  run  whichever  data  sheet  is  specified.  The  choices  are  called 
“Level  3”  (which  actually  means  that  data  can  be  entered  at  Level  2  or  Level  3),  “MRM  Level 
2,”  and  “MRM  Level  1.”  The  labeling  stems  from  the  relationship  to  MRM.3  We  shall  make 
this  more  concrete  later  with  an  example. 

Lor  agile  exploration  of  changed  assumptions,  the  analyst  can  use  the  MRM  Level  2  Data 
sheet,  because  it  has  many  fewer  items  to  specify.  Later,  after  understanding  the  issues  better, 
he  may  revert  to  the  more  detailed  work.  It  may  also  be  that  the  analyst  defers  even  develop¬ 
ing  a  Level  3  description  until  after  considerable  work  has  been  done  at  Level  2  to  sharpen 
appreciation  of  where  additional  detail  is  actually  useful.  This  is  analogous  to  what  many  good 
designers,  architects,  and  analysts  do  routinely:  proceed  top-down. 

It  is  the  analyst’s  responsibility  to  maintain  consistency  among  the  several  data  sheets. 

Translating  Raw  Measures  of  Value  into  Scores 

Any  scorecard  method  (and  any  of  the  classic  decision-analysis  methods,  such  as  those  using 
utility  functions)  requires  that  the  various  measures  of  options’  goodness  be  on  a  common 


3  The  theory  of  MRM  has  evolved  over  the  past  20  years  (Davis  and  Huber,  1992;  Davis  and  Hillestad,  1993;  Bracken, 
Kress,  and  Rosenthal,  1995;  Reynolds,  Natrajan,  and  Srinivasan,  1997;  Natrajan  and  Reynolds,  2001;  Davis  and  Bigelow, 
1998;  Davis  and  Bigelow,  2003;  Yilmaz  and  Oren,  2004).  A  number  of  detailed  applications  to  defense  problems  have  also 
been  published  (Davis,  Bigelow,  and  McEver,  2001;  Davis,  2002b;  National  Academy  of  Sciences,  1996;  Davis,  McEver, 
and  Wilson,  2002). 
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Figure  2.3 

Four  Levels  of  Detail 
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scale.  This  may  be  done  quantitatively  (e.g.,  on  a  0-to-l  scale)  or  visually  with  a  correspond¬ 
ing  discrete  set  of  colors  (e.g.,  red,  orange,  yellow,  light  green,  and  green).  Often,  however, 
the  initial  measures  are  on  heterogeneous  scales.  One  measure  may  be  in  terms  of  probability 
(0  to  1);  another  may  count  something,  such  as  the  number  of  army  divisions  (perhaps  0  to 
100)  or  estimate  the  expected  lifetime  of  a  weapon  system  (in  years).  Such  initial  measures  are 
expressed  in  terms  of  raw  values.  PAT  maps  the  raw  values  into  scores  by  comparing  them  to 
thresholds  and  goals,  as  shown  in  Figure  2.4.  Because  we  seek  ultimately  to  describe  options 
in  discretized  terms,  the  scores  have  lower  and  upper  bounds:  No  matter  how  poor  or  good  a 
raw  value  is,  the  score  will  never  be  lower  than  0  or  higher  than  1.  In  some  instances,  a  higher 
raw  value  is  worse  than  a  lower  value  (e.g.,  higher  risk  is  considered  bad).  In  such  cases,  we  use 
a  simple  variant  of  Figure  2.4  (discussed  in  Chapter  Five). 

Aggregating  Scores 

Level  3  raw  values  are  combined  to  generate  Level  2  scores.  Level  2  raw  values,  as  well 
as  computed  Level  3  scores,  are  combined  to  generate  Level  1  scores.  At  some  point,  the 
Level  1  scores  are  combined  to  generate  a  composite  or  overall  score  called  “effectiveness.”  Cost- 
effectiveness  can  be  calculated,  or  effectiveness  can  be  shown  as  a  function  of  cost  (a  better 
practice).  Figure  2.3  illustrated  the  relationships  schematically,  showing  relative  cost- 
effectiveness  as  the  most  aggregate  of  the  measures  generated  by  PAT. 
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Figure  2.4 

Mapping  of  Raw  Values  into  Scores 


Score 


Diagrams  such  as  Figure  2.3  indicate  with  arrows  that  different  measures  combine  to 
generate  the  score  of  a  higher-level  measure,  but  they  do  not  say  how  they  combine.  It  is  fre¬ 
quently  assumed  in  commercial  decision-analysis  tools  and  in  introductory  courses  in  decision 
analysis  that  factors  combine  via  linear  weighted  sums.  This  is  so  common  that  it  has  affected 
our  vocabulary,  as  when  we  refer  to  the  relative  “weight”  of  different  input  variables. 

Linear  weighted  sums  are  often  convenient  and  adequate,  but  they  can  be  quite  mislead¬ 
ing,  because  in  the  real  world,  the  combining  rules  are  nonlinear.  Do  we  want  to  be  healthy  or 
wealthy?  No  amount  of  health  can  compensate  for  extreme  poverty  and  no  amount  of  wealth 
can  compensate  for  extreme  illness.  Overall  utility  is  not  a  “sum”  of  scores  for  health  and 
wealth,  but  something  more  complicated.  Similarly,  in  designing  a  system  or  an  investment 
program,  the  value  of  the  whole  may  be  essentially  zero  unless  each  of  various  critical  compo¬ 
nents  is  sufficiently  good. 

Because  of  such  considerations,  PAT  provides  five  built-in  methods  for  establishing  scores 
and  aggregating  upward.  Ideally,  only  one  such  method  would  be  needed,  but  theory  and 
experience  tell  us  that  alternatives  are  needed — even  more  alternatives  than  the  five  built-in 
methods.  Thus,  we  have  also  allowed  for  extensibility,  as  discussed  later. 

The  five  built-in  methods  are  defined  in  Table  2.1,  using  the  concepts  of  thresholds,  goals, 
and  nonlinearity.  Chapter  Five  provides  more  details  on  all  five  methods.  Of  the  five,  the  first 
three  are  the  core  methods,  referred  to  as  the  Thresholds,  Weakest  Link,  and  Weak  Threshold 
methods.  The  default  method  (Thresholds)  characterizes  a  given  aggregate  measure’s  score  as 
zero  if  any  of  its  submeasures’  scores  are  below  analyst-specified  thresholds.  That  is,  a  Level  1 
measure  is  zero  by  this  method  if  any  of  its  Level  2  submeasures  are  below  the  threshold;  simi¬ 
larly,  a  Level  2  measure  is  zero  by  this  method  if  any  of  its  Level  3  submeasures  are  below  the 
threshold.  This  enforces  the  concept  that  a  system  fails  if  any  of  its  critical  components  fail. 
The  method  is  appropriate  if  the  submeasures  happen  to  be  individually  critical.  The  Weakest 
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Table  2.1 

Core  Built-In  Aggregation  Methods 


Method 

Component- 
Measure  Scores 

Aggregate-Measure 

Scores 

Overall  Effectiveness 
(across  all  measures) 

Comment 

Thresholds 

See  Figure  2.3 

0  if  any  raw  value  does 
not  reach  threshold; 
otherwise,  a  weighted 
sum  of  component- 
measure  scores 

Weighted  sum  of 
measures'  scores 

Appropriate  if 
component  measures 
represent  critical 
components  of 
capability 

Weakest  Link8 

See  Figure  2.3 

Minimum  of  component- 
measure  scores 

Minimum  of  measures' 
scores 

Appropriate  if  both 
component  measures 
and  measures  are 
individually  critical 

Weak  Thresholds 

See  Figure  2.3 

Weighted  sum  of 
component-measure 
scores 

Weighted  sum  of 
measures'  scores 

Appropriate  if 
thresholding  is 
valuable  but  not  all 
component  measures 
or  measures  are 

critical 

Goals 

Weighted  fraction  of  the 
component-measure 
goals  achieved  by 
option 

Weighted  sum  of 
measures'  scores 

Rankings 

Borda  ranking*3 

Borda  ranking*3 

aThis  option  was  introduced  in  the  DynaRank  system  (Hillestad  and  Davis,  1998). 
bA  single-winner  election  method  in  which  voters  rank  candidates  in  order  of  preference. 


Link  method  is  similar  but  even  more  stringent.  It  assesses  the  aggregate  score  to  be  the  lowest 
of  the  contributing  scores  and  assesses  overall  effectiveness  as  the  lowest  of  the  measure  scores.4 
The  third  method  (Weak  Thresholds)  is  less  draconian.  Both  measure-level  scores  and  overall 
effectiveness  are  simply  weighted  sums.  A  contributing  factor  is  scored  zero  if  it  does  not  reach 
its  threshold  value,  but  the  aggregate  score  is  the  sum  of  the  factors’  scores  (rather  than  zero). 
This  method  is  suitable  if  the  contributing  measures  are  not  individually  critical  but  it  seems 
important  to  impose  thresholding. 

These  methods  are  effective  heuristics  that  are  well  understood  by  decisionmakers.  A 
tough  commander  or  manager,  for  example,  may  consider  a  unit  to  have  “failed  inspection,” 
even  if  the  unit  did  rather  well  in  many  respects. 

The  other  two  aggregation  methods  (goals  and  rankings)  are  useful  in  certain  cases,  as 
discussed  in  Chapter  Five. 


Extensibility:  Allowing  Custom  Aggregation  Methods 

In  several  of  our  applications  of  PAT,  we  have  found  it  necessary  to  aggregate  results  in  ways 
that  do  not  lend  themselves  well  to  the  built-in  options.  This  can  be  done  straightforwardly, 
using  standard  features  of  Excel  and  modest  amounts  of  mathematics. 


4  This  rule  was  introduced  in  Hillestad  and  Davis,  1998. 
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Hint  (one  that  will  be  meaningful  only  to  someone  who  is  ready  to  actually  use 
PAT):  One  approach  when  calculating  an  aggregation  from  a  given  level  is  (1)  add 
a  new  measure  at  that  level  called  "calculated  score";  (2)  define  option  values  for  that 
measure  with  an  equation  referring  to  other  data  (e.g.,  to  the  product  of  previous 
measures  at  the  same  level);  (3)  set  the  weights  of  the  other  measures  to  be  very  small 
but  not  zero  (e.g.,  0.0001)  and  the  weight  of  the  calculated  score  to  be  1;  (4)  use  any 
of  the  three  core  scoring  methods  in  Table  2.1.  The  result  will  be  that  the  aggregate 
score  will  be  the  calculated  value,  but  the  analyst  will  still  see  the  values  of  the 
component  factors  when  using  PAT's  drilldown  feature  as  described  in  Chapter  Three. 

NOTE:  When  extending  PAT's  functionality  by  entering  equations,  users  should  ordinarily  use 
"absolute  references"  (with  $  signs)  so  that  the  formulas  will  be  automatically  adjusted  if  they  change 
PAT's  structure  by,  e.g.,  adding  a  column  or  row.  Checking  is  important.  Users  should  also  avoid — or 
at  least  be  very  cautious  about — using  array  formulas  within  PAT  if  further  changes  in  structure  are 
likely. 


Navigation  and  Architecture:  Inputs  and  Outputs 

We  now  provide  an  overview  of  how  one  navigates  within  PAT,  the  inputs  that  must  be  pro¬ 
vided,  and  the  outputs  it  generates.  Subsequent  chapters  will  describe  inputs  and  outputs  in 
much  more  detail. 

Architecture  and  Navigation 

Since  PAT  is  implemented  in  Excel,  it  uses  a  spreadsheet  paradigm  for  entering  data  and  gen¬ 
erating  familiar  kinds  of  tables  and  charts.  In  basic  terminology,  a  PAT  file  is  an  Excel  “work¬ 
book”  that  contains  multiple  “worksheets.”  (In  this  report,  we  use  the  terms  “worksheet”  and 
“sheet”  interchangeably.)  The  user  navigates  among  worksheets  in  three  ways:  (1)  by  using  the 
Go  To  Sheet  menu  item  at  the  top  of  the  program’s  window,5  (2)  by  clicking  tabs  along  the 
bottom  of  the  program’s  window  (the  standard  method  in  Excel);  or  (3)  by  clicking  buttons 
that  appear  in  some  sheets  (again,  a  standard  method  in  Excel).  The  first  option  is  often  the 
easiest.  Figure  2.5  shows  where  the  menu  is  found  (on  any  PAT  sheet);  Figure  2.6  shows  tabs  at 
the  bottom  of  the  Excel  window;  and  Table  2.2  shows  the  menu  of  built-in  worksheets. 

Figure  2.5 

Menu  of  Input  and  Output  Sheets 


□  Microsoft  Excel  -  Annotated  Example2b 
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5  In  Excel  2007,  the  Go  To  Sheet  menu  is  accessed  under  the  Other  Add-Ins  menu. 
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Figure  2.6 

Tabs  for  Moving  Among  Sheets 


.Tabs 


►i  \  Summary  /  Level  2  Drilldown  /  Level  3  Drilldown  /  Level  1  Data  /  Level  2  Data  /  Level  3  Data  /  MRM  1 1  < 


Table  2.2 

PAT  Output  and  Input  Sheets 


Output  Sheets 

Input  Sheets 

Tables  (e.g.,  Scorecards) 

Level  1  Data 

Summary 

Level  2  Data 

Level  2  Drilldown 

Level  3  Data 

Level  3  Drilldown 

MRM  Level  1  Data 

Selected  Details 

MRM  Level  2  Data 

Rankings  Table 

Cost  Data 

Graphics 

Perspectives 

Scatter  Plot 

Template  Builder 

Spider  Charts 

Multimeasure  Spider  Charts 

Cost  Charts 

Although  convenient,  the  Go  To  Sheet  menu  omits  custom  sheets  that  the  user  may 
have  added  for  extra  data,  specialized  calculations,  or  notes.  In  contrast,  all  of  the  sheets 
have  corresponding  tabs  along  the  bottom.  The  standard  Excel  way  to  navigate  among 
sheets  is  by  clicking  those  tabs.  Table  2.3  shows  the  default  left-to-right  order  of  tabs,  but 
users  may  have  other  tabs  corresponding  to  application-specific  worksheets  and  will  in 
any  case  find  it  convenient  to  reorder  the  tabs  by  moving  them  around.  Thus,  the  order 
in  Table  2.3  may  not  apply.  A  Notes  sheet  is  optional,  something  we  suggest  that  the  user 
add  as  a  custom  sheet  for  keeping  track  of  changes  and  subtleties  and  for  maintaining  con¬ 
figuration  control  (i.e.,  assuring  that  collaborators  are  using  the  same  version  of  PAT). 

In  a  spreadsheet  program  such  as  PAT,  “architecture”  is  largely  indicated  by  the  choice  of 
worksheets  and  the  relationships  among  them  (e.g.,  Level  2  Drilldown  provides  more  detail  on 
some  aspects  of  the  results  shown  in  the  Summary  sheet). 

Inputs  and  Outputs 

With  this  background,  let  us  review  schematically  PAT’s  key  outputs.  This  will  provide  a  sense 
of  what  PAT  does  before  we  get  into  the  details.  Figure  2.7  is  the  schematic  diagram  of  our 
topmost  display,  the  Summary  sheet. 
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Table  2.3 

Default-Ordered  Listing 
of  Tabs  for  PAT  Sheets 


Welcome  Screen 
Summary 
Level  2  Drilldown 
Level  3  Drilldown 
Scatter  Plot 
Spider  Charts 
Cost  Charts 
Rankings  Table 
Level  1  Data 
Level  2  Data 
Level  3  Data 
MRM  Level  1  Data 
MRM  Level  2  Data 
Selected  Details 
Cost  Data 
Perspectives 
Template  Builder 
Dropdown 


Figure  2.7 

Schematic  of  Summary  Sheet 
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At  the  top  of  the  sheet  are  some  controls.  At  the  bottom  of  the  sheet  is  a  translation 
from  color-coding  into  numbers.  In  the  main  portion  of  the  sheet,  rows  represent  investment 
options.  Each  option  is  scored  by  different  criteria  or  measures  represented  by  the  columns. 
The  hrst  block  of  assessments  is  the  color-coded  scorecard  of  Level  1  measure  values  (A,  B,  C). 
Moving  rightward,  blocks  contain  numeric  values,  such  as  of  selected  measures  that  the  analyst 
wishes  to  highlight,  cost  data,  and  effectiveness  and  cost-effectiveness  values.  Ordinarily,  the 
user  will  have  only  a  portion  of  the  Summary  sheet  visible  (e.g.,  the  scorecard  portion). 

The  primary  challenges  in  working  with  such  high-level  depictions  are  assuring  that  they 
frame  the  problem  well,  highlighting  the  right  considerations,  and  assuring  that  the  depictions 
reflect  assessments  based  on  solid  analysis  with  a  good  audit  trail.  The  analysis  may  be  based  on 
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information  from  capability  models,  structured  subjective  judgment  sources,  expert  judgment 
based  on  detailed  studies,  or  other  sources. 

Users  should  have  the  ability  to  understand  why  a  given  cell  of  the  Summary  sheet  is 
red,  rather  than  green  (bad,  rather  than  good)  or  where  various  numbers  came  from.  This 
can  be  done  to  a  significant  extent  within  PAT  by — for  a  particular  measure — drilling  down 
or  zooming  (we  use  the  terms  interchangeably)  to  sheets  describing  matters  at  Level  2  and 
Level  3.  Figure  2.8  shows  this  schematically,  suppressing  all  aspects  of  the  sheets  except  the 
scorecards  at  Level  1,  Level  2,  and  Level  3.  In  the  example,  the  user  drills  down  on  Measure  2 
of  the  Summary  chart,  discovering  that  it  is  based  on  two  Level  2  measures.  If  he  drills  down 

Figure  2.8 

Schematic  of  Drilldown  (Zoom) 
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further,  on  the  second  of  the  two  Level  2  measures,  he  finds  that  Measure  2  is  based  on  calcu¬ 
lations  using  three  Level  3  measures.  In  most  cases,  this  drilling  down  provides  an  adequate 
visual  explanation  of  results.  However,  such  is  not  always  the  case,  and  it  may  be  necessary 
to  study  more  analytically  detailed  results  from  capability  models  and  the  equations  used  to 
generate  aggregations. 

PAT  also  generates  numerous  graphics,  which  can  be  useful  in  the  course  of  analysis  and 
in  the  presentation  of  results.  Figure  2.9  illustrates  several  types  of  built-in  graphics:  a  line 
chart,  a  spider  chart,  and  a  scatter  plot.  Examples  of  each  type  of  chart  will  be  discussed  in 
later  chapters.  Because  a  PAT  file  is  an  Excel  spreadsheet,  the  user  can  also  create  and  custom¬ 
ize  charts,  using  Excel’s  standard  features. 

With  this  quick  overview,  let  us  now  turn  to  a  more  detailed  discussion.  Chapters  Three 
and  Four  describe  input  and  output  sheets  at  a  reference-manual  level  of  detail.  Chapter  Five 
then  provides  detail  on  methodology,  presenting  equations  and  definitions  for  the  most  general 
cases  treated  by  PAT.  Chapter  Six  illustrates  marginal  analysis. 

Readers  who  prefer  to  move  directly  into  “doing”  rather  than  reading  may  wish  to  use  the 
QuickStart  example  in  Appendix  A. 

Figure  2.9 

Sample  Output  Displays  from  PAT 
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CHAPTER  THREE 


PAT  Input  Worksheets 


This  chapter  describes  each  PAT  input  worksheet1  and  shows  either  a  screen  capture  or  a  sche¬ 
matic  of  each.  Names  of  the  worksheets  are  in  italics  throughout  the  report,  except  in  headings. 


Template  Builder 

Users  will  ordinarily  start  their  work  with  PAT  by  filling  out  the  information  in  a  tool  called 
Template  Builder.  Here  they  will  specify  the  structure  of  most  sheets,  such  as  the  names  of  the 
Level  1,  Level  2,  and  Level  3  measures;  the  names  of  the  options;  the  time  period;  cost  catego¬ 
ries  and  expenditure  items;  and  control  variables.  Template  Builder  then  generates  various  other 
sheets  appropriately,  at  which  time  the  user  specifies  the  option-dependent  data  and  fine-tunes 
some  of  the  controls.  It  is  akin  to  filling  out  an  outline  and  having  the  computer  generate  an 
entire  book  with  chapter  and  section  structure  but  no  content.  Despite  the  centrality  of  Tem¬ 
plate  Builder  for  structuring  PAT  in  practice,  we  defer  its  discussion  until  later  in  this  chapter 
because  serious  users  should  first  understand  PAT’s  underlying  architecture.  This  will  improve 
intuition  for  what  can  and  cannot  be  done  and  will  help  them  diagnose  problems  that  may 
arise  because  of  erroneous  data  entries. 


Level  1  Data 

Figure  3.1  shows  the  Level  1  Data  sheet  for  a  simple  case  with  only  three  measures.  These  data 
are  reflected  on  the  Summary  sheet.  The  data  to  be  inputted  are  (1)  the  names  of  the  measures 
to  be  displayed,2  (2)  optional  comments,  and  (3)  the  relative  weights  of  the  measures  when 
calculating  overall  option  effectiveness.  As  discussed  in  Chapter  Four,  if  a  user  “mouses  over” 
(i.e.,  passes  the  cursor  over)  a  measure’s  name  in  the  Summary  sheet,  a  pop-up  will  show  any 
comments  (e.g.,  a  cryptic  definition)  and  weights.  In  this  case  the  only  comment  is  to  define 
Measure  1  as  “Effectiveness  in  Scenario  1;  measured  by  outcome.”  After  entering  data,  the  user 
can  click  the  Modify  Summary  button  to  have  the  data  take  effect. 


1  As  with  most  modern  tools,  the  distinction  between  input  and  output  sheets  is  blurred,  because  some  inputs  can  be 
changed  in  output  sheets.  This  undercuts  architectural  clarity  and  can  be  confusing  but  is  quite  useful  in  practice. 

2  Once  entered,  these  cannot  be  changed  in  Level  1  Data  alone  but  can  be  changed  using  Find  and  Replace. 
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Figure  3.1 
Level  1  Data 


Modify  Summary 


Measure 

Comment 

Weight 

Measure  1 

Effectiveness 
in  Scenario  1; 
measured  by 
outcome 

1.00 

Measure  2 

1.00 

Measure  3 

1.00 

Although  this  is  the  official  input  sheet  for  Level  1,  users  will  usually  employ  Template 
Builder  to  create  the  structured  input  and  output  sheets,  and  they  will  usually  specify  the 
weights  of  Level  1  measures  in  the  Summary  sheet  (even  though  that  is  nominally  an  output 
sheet). 


Level  2  Data 

The  values  of  Level  1  measures  (i.e.,  summary  measures)  are  determined  by  data  at  Levels  2 
and  3.  Figure  3.2  shows  the  Level 2  Data  sheet  for  our  simple  case  with  three  measures,  each 
with  two  submeasures.  That  is,  each  Level  1  measure  depends  on  two  Level  2  measures. 

Names  of  Measures  and  Options 

The  Level  1  measure  names  are  entered  as  column  headers  precisely  as  shown  (one  row  for  the 
Level  1  measure’s  name;  one  row  for  each  of  the  Level  2  measures’  names,  including  items 
called  “Warning”).  The  names  of  the  investment  options  are  entered  as  row  headers.  All  such 
names  must  be  consistent  across  workbook  sheets.  For  example,  the  Level  1  measure  names 
must  agree  with  those  in  the  Level  1  Data  sheet. 


Hint:  Editing  names,  which  is  commonly  important  to  improve  intuitive  clarity,  is  best  done 
by  using  Excel's  Find  and  Replace  function  (applying  it  to  the  workbook  rather  than  to  the 
current  sheet),  rather  than  attempting  to  type  consistently  across  sheets. 


Data  on  Option  Effectiveness 

The  lower  part  of  Figure  3.2  (below  the  yellow  divider)  is  filled  in  with  Level  2  data  for  each 
investment  option  and  measure.  Most  of  the  data  can  be  directly  inputted  at  this  level,  but 
Measure  2.2’s  italicized  values  indicate  that  they  are  calculated  from  Level  3  information.  If 
the  analyst  types  over  the  italicized  numbers,  the  calculated  numbers  will  be  regenerated  in 
the  next  PAT  run,  overwriting  the  “corrections.”  Thus,  any  changes  to  Measure  2.2’s  data 
must  be  made  at  Level  3.  Any  entry  in  the  Warning  column  for  a  particular  option  should 
be  text,  as  in  the  example  shown  for  Measure  1 .  Such  warnings  cause  little  flags  to  appear  in 
cells  of  the  Summary  scorecard.  They  appear  in  pop-ups  if  the  user  mouses  over  the  flags  (see 
Chapter  Four). 

The  data  on  the  options  may  come  from  a  capabilities  model  or  other  sources,  including 
subjective  judgment  or  detailed  studies. 
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Parameters  Specifying  the  Nature  of  Scoring  and  Aggregation 

The  rows  at  the  top  of  Figure  3.2  (above  the  yellow  divider)  specify  the  control  parameters 
needed  to  aggregate  from  Level  2  to  Level  1 .  How  and  whether  a  given  control  parameter  is 
actually  used  depends  on  the  scoring  and  aggregation  methods  chosen,  which  will  be  discussed 
later.  These  parameter  values  can  be  left  blank  for  Warning  columns;  PAT  ignores  any  values 
that  are  entered. 

Figure  3.2 
Level  2  Data  Sheet 
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The  scoring  and  aggregation  functions  depend  on  parameters  such  as  goal  values.  Four 
rows  in  the  Level  2  Data  sheet  set  those  parameters: 

•  Weight  of  Level  2  Measure  in  Scoring  Functions  (0  to  1).  The  relative  weights  of  the  Level  2 
measures  when  calculating  the  Level  1  scores  (those  used  for  the  Summary  sheet).  A 
weight  of  zero  means  that  the  Level  2  measure  is  not  considered.  Thus,  a  Level  2  measure 
can  be  built  in  but  then  toggled  on  or  off  as  appropriate. 

•  High  or  Low  Values  Desired ?  Either  High  or  Low  (capitalization  matters).  This  row  allows 
users  to  specify  that  for  some  measures  (e.g.,  probability  of  failure),  more  is  worse  rather 
than  better.  If  Low  is  chosen  (e.g.,  as  with  Measure  1.2),  scoring  uses  a  variant  of  Figure 
2.2.  This  choice  of  directionalities  applies  only  at  the  lowest  level  of  data  entry;  PAT  re¬ 
scales  higher  measures  to  be  on  a  0-to-l  scale,  with  1  being  better. 


Hint:  This  dictates  care  in  naming.  Lowest-level  entries  for  various  types  of  risk  might 
be  defined  so  that  low  numbers  are  good.  Aggregations,  however,  because  they  will  be 
rescaled  by  PAT,  should  be  named  something  like  "Confidence"  or  "Risk  Mitigation"  to 
avoid  semantic  confusion. 


•  Threshold  Value.  The  threshold  values  described  in  Chapter  Two  for  the  Thresholds, 
Weakest  Link,  and  Weak  Thresholds  scoring  methods. 

•  Goal  Value.  The  goal  value  used  in  all  scoring  functions  except  Rankings.  Where  high 
values  are  desired,  the  goal  value  cannot  be  lower  than  the  threshold  value.  Where  low 
values  are  desired,  the  goal  value  cannot  be  higher  than  the  threshold  value. 
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Effectiveness  and  cost-effectiveness  calculations  aggregate  across  measures  and  need  to 
operate  on  a  common  scale.  This  is  accomplished  by  setting  two  further  parameters: 

•  Level  2  Measure  Score  for  Threshold  Value  (0  to  1).  Scores  assigned  to  Level  2  measures  if 
their  thresholds  are  just  reached  (for  scoring  methods  that  incorporate  thresholds). 

•  Level  2  Measure  Score  for  Goal  Value  (0  to  1).  Scores  assigned  if  goal  values  are  reached 
or  exceeded.  For  scoring  methods  with  thresholds,  the  effectiveness  score  for  a  value  that 
falls  between  the  threshold  and  the  goal  value  is  interpolated  linearly  (Figure  2.2).  It  is 
good  practice  to  set  this  value  to  1  for  all  measures. 

Warning  Comments 

Each  Level  1  measure  may  reflect  a  special  Level  2  measure  called  Warning.  The  data  in  the 
Warning  columns  should  be  text.  If  a  given  cell  has  a  warning,  the  Summary  display  will  have 
a  small  flag  in  the  top  right  corner  of  the  corresponding  cell.  Mousing  over  that  cell  will  bring 
the  warning  message  up.  This  mechanism  is  a  convenient  way  of  flagging  results  that  depend 
on  reasonable  but  worrisome  assumptions.  In  a  defense-planning  analysis,  such  a  flag  might  be 
“Assumes  at  least  one  week’s  actionable  strategic  warning.” 

The  Modify  Summary  Button 

After  making  changes  in  data  at  any  level,  the  user  should  click  the  Modify  Summary  button 
(top  left)  or  go  to  the  Summary  sheet  and  select  Update  Summary  to  have  the  changes  take 
effect. 


Cautionary:  If  any  Level  2  measure  for  a  particular  Level  1  measure  is  calculated  from 
Level  3  data,  any  Level  2  measures  inputted  directly  should  also  be  given  raw  values 
from  0  to  1.  PAT  reports  the  calculation  from  Level  3  using  the  0-to-1  range,  and  visual 
explanations  such  as  the  Level  2  Drilldown  sheet  become  non-intuitive  if  the  various 
Level  2  measures  are  on  different  scales. 

Hint:  Entering  data  with  decimal  points  can  be  more  tedious  and  prone  to  error  than 
entering  whole-integer  numbers.  The  user  can  enter  the  option-specific  data  with 
whole  integers  and  then  scale  them  down  by  (1)  entering  0.1  in  a  cell  outside  the  array, 
(2)  cutting,  (3)  selecting  the  range  of  cells  to  be  scaled,  and  (4)  selecting  Edit/Paste  Special/ 
Multiply  from  the  Excel  menu.  If  that  menu  option  does  not  appear,  use  Tools/Add-lns  to 
activate  some  optional  features  of  Excel.  Be  sure  not  to  leave  any  stray  items  in  the  sheet 
(e.g.,  the  0.1). 


Level  3  Data 

Figure  3.3  shows  the  Level  3  Data  sheet  for  our  simple  example  in  which  only  Measure  2  has 
Level  3  data  (Measures  2.2.1  and  2.2.2).  The  sheet  is  very  similar  to  that  for  Level  2.  Although 
small  in  this  example,  it  is  often  quite  large  in  applications — it  may  have  tens  of  columns. 
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Figure  3.3 
Level  3  Data  Sheet 
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At  the  top  left  of  the  sheet  is  a  button  called  Modify  Level  2  Data  Sheet  (used  to  recalcu¬ 
late  the  values  of  Level  2  information  generated  from  Level  3).  The  tiny  box  with  a  checkmark 
should  remain  checked.3 

Some  Level  3  data  can  be  changed  in  output  sheets,  but  the  user  should  be  sure  to  click 
appropriate  buttons,  such  as  Modify  Level  2  Data  Sheet,  so  that  the  changes  will  take  effect. 
Arguably,  this  functionality  (of  changing  inputs  in  an  output  sheet)  should  be  avoided  because 
of  the  potential  for  confusion. 


MRM  Level  1  and  Level  2  Data 

The  MRM  Level  1  Data  and  MRM  Level  2  Data  sheets  allow  the  user  to  have  a  low-detail 
mechanism  for  inputting  data,  so  that  fewer  changes  are  necessary  when  doing  sensitivity 
analysis  or  more  general  exploratory  analysis.  This  can  be  quite  useful,  but  the  user  should 
remember  that  these  data  sheets  are  distinct  from  the  nominal  data  provided  in  the  Level  2 
Data  and  Level  3  Data  sheets.  The  analyst  must  maintain  such  consistency  as  is  needed. 

This  MRM  functionality  is  valuable  to  the  analyst  who  wishes  to  work  at  a  low  level  of 
resolution  until  everything  is  “making  sense,”  at  which  point,  more  detail  and  more  care  in 
specifying  data  may  be  appropriate.  This  is  standard  in  good  top-down  analysis  and  in  soft- 


3  This  option  affects  certain  displays  if  a  measure’s  raw  value  is  below  its  threshold.  It  is  best  ignored.  However,  a  checked 
box  means  that  a  measure  with  a  score  well  below  its  threshold  will  get  the  same  score  (0)  as  it  would  have  if  it  had  just 
reached  the  threshold.  An  unchecked  box  assigns  a  score  of -1  in  the  former  case. 
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ware  engineering.  Without  the  MRM  option,  making  even  what  seem  like  simple  changes  may 
require  entering  many  low-level  data  values.  That  is  both  tedious  and  an  invitation  to  error. 

More  important  ultimately  (but  more  controversial  among  analysts  who  like  to  argue 
about  such  things)  is  the  fact  that  the  MRM  design  encourages  thinking  top-down  and  elimi¬ 
nates  the  requirement  to  enter  what  may  be  spurious  detail.  In  many  instances,  the  informa¬ 
tion  available  for  the  assessment  of  a  particular  measure  is  inherently  of  low  resolution.  For 
example,  if  expert  consultants  with  years  of  managerial  experience  are  asked  to  judge  the  risk 
of  some  alternative  programs,  they  may  be  able  to  do  so  quickly  and  well.  However,  if  they  are 
asked  to  break  down  their  judgments — listing  components  and  subcomponents  of  risk,  and 
then  estimating  values  and  probabilities  of  those  risks  individually — the  quality  of  the  assess¬ 
ment  may  worsen  rather  than  improve.  The  reasons  for  this  are  many.  One  reason  is  that  many 
things  can  go  wrong,  and  experts  often  smell  the  potential  for  problems  without  being  able  to 
predict  which  particular  problems  will  arise.  If  they  are  forced  to  itemize,  they  may  omit  some 
of  the  possibilities.  Another  theoretical  problem  is  that  the  errors  made  in  low-level  (high- 
resolution)  subjective  estimates  do  not  propagate  upward  nicely.  An  aggregate-level  assessment 
may  be  more  accurate  than  the  result  of  calculations  based  on  many  low-level  assessments. 

For  brevity,  we  do  not  show  examples  of  the  MRM  Level  1  and  MRM  Level  2  Data  sheets 
here.  The  former  is  trivial — one  merely  inputs  the  “answers”  that  are  to  be  displayed  in  the 
Summary  display.  The  latter  looks  like  and  operates  like  the  Level  2  Data  sheet. 


Hint:  Two  usage  scenarios  are  worth  mentioning.  In  the  first,  the  analyst  is  thinking  top- 
down  and  doesn't  want  to  specify  details.  Like  a  designer  or  system  engineer,  he  leaves 
"markers"  or  "stubs"  by  entering  the  names  of  Level  2  and  Level  3  measures,  but  instead 
of  filling  in  the  detail,  he  creates  a  separate  MRM  Level  2  Data  sheet  and  fills  that  out. 
He  does  initial  work  at  that  level  (invoking  that  data  sheet  in  the  Summary  sheet's  Multi- 
Resolution  Modeling  menu).  When  the  time  comes,  he  reverts  to  using  the  data  sheet 
with  both  Level  2  and  Level  3  data  and  fills  in  details  as  needed. 

In  the  second  scenario,  the  analyst  begins  bottom-up,  working  with  Level  3  detail. 
At  some  point,  he  finds  it  necessary  to  be  more  agile  and  to  think  at  a  higher  level — 
perhaps  when  exploring  uncertainty  or  responding  to  aggregate-level  "What  if  .  .  .?" 
questions.  He  copies  his  previous  Level  2  information  from  the  Level  2  Drilldown  sheet 
into  the  MRM  Level  2  Data  sheet,  invokes  MRM  Level  2  data  with  the  Multi-Resolution 
Modeling  menu,  and  proceeds.  When  he  is  done,  he  may  revert  to  the  detailed  analysis. 
He  may  find,  however,  that  some  Level-3-to-Level-2  calculations  were  spurious  because  of 
hopelessly  ambiguous  data  or  uncertain  phenomena.  Dispensing  with  such  spurious  (and 
pretentious)  detail  would  then  be  desirable.  In  other  cases,  the  detail  is  essential. 


Using  MRM  Level  1  data  is  unusual  but  can  be  useful  for  experimenting  with  story  lines 
and  displays.  Using  it  will  specify  the  colors  that  appear  in  the  Summary  scorecard. 

Some  of  the  items  in  MRM  data  sheets  can  be  changed  in  output  sheets,  but  the  user 
should  then  click  appropriate  buttons  to  trigger  updating.  It  is  best  to  make  the  changes  in  the 
input  sheet  and  click  the  appropriate  Modify  button.  And,  as  mentioned  earlier,  it  is  arguably 
dangerous  to  enter  inputs  in  output  sheets,  because  of  the  potential  for  confusion. 
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Cost  Data 

Establishing  Cost  Structure 

PAT  allows  the  user  considerable  latitude  in  representing  cost  information  about  various 
options.  The  choices  include: 

•  Time  frame  (e.g.,  six-year,  20-year,  or  forever  costs) 

•  Categories  of  cost  (e.g.,  R&D,  acquisition,  O&S) 

•  Items  (the  classes  of  investment  items,  such  as  hardware  and  software,  or  ships,  aircraft, 
and  tanks) 

•  Discount  rate  (representation  of  inflation,  or  of  inflation  plus  the  real  discount  rate  as 
needed  in  present-value  calculations). 

Figure  3.4  illustrates  the  format  of  the  Cost  Data  sheet.  For  compactness,  this  example 
has  only  a  three-year  time  frame  and  two  types  of  investment  items.  A  more  realistic  example 
would  require  scores  of  columns  and  rows. 

To  specify  cost  structure  and  data  starting  with  a  blank  Cost  Data  sheet,  the  user  should 
specify  one  block  of  columns  for  each  cost  category.  Each  block  should  have  the  same  number 
of  columns  with  the  same  range  of  years.  One  block  of  rows  should  be  entered  for  each  option. 
Each  block  should  have  the  same  set  of  rows  representing  different  cost  items  over  the  same 
time  period.  Each  block  of  columns  and  rows  should  be  separated  from  the  next  by  one  column 
or  row,  respectively. 

The  analyst  is  responsible  for  entering  appropriate  data  such  as  current-year  (also 
called  then-year)  or  real  (inflation-corrected)  costs.  For  present-value  calculations  of  cost- 
effectiveness,  a  real  discount  factor  can  be  specified  here  or  in  the  Summary  sheet.  That  will 
override  data  in  the  data  sheet  when  making  calculations  for  the  Summary  sheet.  That  is,  if  the 
Cost  Data  sheet  has  constant-dollar  entries,  but  the  Summary  sheet  specifies  a  discount  rate  of 
3  percent,  the  dollar  values  listed  in  the  Summary  sheet  and  used  for  cost-effectiveness  calcu¬ 
lations  will  be  in  present-value  terms  assuming  that  the  value  of  money  is  3  percent  on  top  of 
inflation. 

An  Easier  Approach  to  Structuring  Cost  Data 

It  can  be  tedious  to  set  up  the  data  structure  in  the  Cost  Data  sheet,  so  the  recommended 
approach  is  to  specify  the  structure  in  Template  Builder,  which  will  then  fill  out  the  sheet 
except  for  the  costs  of  the  options  themselves.  Simplification  is  also  desirable  in  many  cases,  as 
shown  below. 

Customized  Cost  Calculations 

It  will  sometimes  be  desirable  to  customize  cost  data  or  cost  calculations  in  ways  not  antici¬ 
pated  by  PAT’s  built-in  options  or  to  juxtapose  calculations  for  different  assumptions  (e.g., 
different  rates  of  inflation  or  discount  rate,  or  different  horizons  short  of  infinity  for  calculat¬ 
ing  present  values).  That  can  be  accomplished  in  a  separate  worksheet  that  may  draw  from  the 
Cost  Data  sheet  (Davis  et  ah,  2008).  Results  can  be  displayed  in  the  Summary  sheet,  using  its 
custom  cost  columns  or  in  the  worksheet  itself. 


Figure  3.4 
Cost  Data  Sheet 


Modify  Summary 

Discount  Rate  (Rel  to  First  Year 

0 

id 

on  Cost  ($M) 

Operations  Cost($M) 

R&D  Cost  ($M) 

Acquisiti 

Option  A 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  A 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  A 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Hardware 

90 

28 

0 

0 

0 

0 

Hardware 

0 

0 

1,500 

200 

100 

20 

Hardware 

0 

0 

0 

10 

120 

120 

Software 

30 

28 

28 

10 

4 

3 

Software 

0 

0 

100 

50 

20 

10 

Software 

0 

0 

0 

30 

30 

30 

R&D  Cost  ($M) 

Acquisition  Cost($M) 

Operations  Cost($M) 

Option  B 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  B 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  B 

2,010 

2,011 

2,012 

2.013 

2,014 

2,015 

Hardware 

100 

30 

0 

0 

0 

0 

Hardware 

0 

0 

1,200 

200 

100 

20 

Hardware 

0 

0 

0 

10 

100 

100 

Software 

30 

30 

30 

10 

4 

3 

Software 

0 

0 

100 

50 

20 

10 

Software 

0 

0 

0 

10 

10 

10 

R&D  Cost  (SM) 

Acquisition  Cost($M) 

Operations  Cost($M) 

Option  C 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  C 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Option  C 

2,010 

2,011 

2,012 

2,013 

2,014 

2,015 

Hardware 

70 

20 

0 

0 

0 

0 

Hardware 

0 

0 

100 

50 

30 

10 

Hardware 

30 

35 

40 

45 

50 

55 

Software 

30 

20 

20 

10 

4 

3 

Software 

0 

0 

50 

30 

10 

5 

Software 

10 

15 

20 

25 

30 

35 

24  RAND's  Portfolio  Analysis  Tool  (PAT):  Theory,  Methods,  and  Reference  Manual 


PAT  Input  Worksheets  25 


Simplifications 

It  is  often  desirable  for  the  analyst  to  take  shortcuts  in  dealing  with  costs,  especially  early  in  a 
project.  He  may  wish  to  think  simply  in  terms  of  total  costs — over  all  categories  and  over  all 
the  years  of  interest.  Further,  when  he  conducts  sensitivity  analyses,  he  may  wish  to  change 
only  one  cost  number  per  option. 

PAT  has  no  “multiresolution  data  entry”  option  for  costing,  but  the  simplification  is  easy. 
Either  in  the  Cost  Data  sheet  directly  or  via  Template  Builder,  the  analyst  can  specify  having 
only  a  single  cost  category  such  as  “Total  Cost  ($M)”  and  the  only  investment  item  may  be 
“Stuff.” 


Hint:  An  additional  worksheet  can  be  created  to  enter  detailed  cost  information,  for  use 
when  needed.  Later,  the  information  might  be  entered  into  Cost  Data  with  a  revised 
format. 


The  level  of  detail  used,  then,  is  very  much  up  to  the  analyst  and  can  be  changed  in  the  course 
of  a  study. 


Perspectives 

The  Basic  Concept  of  Perspectives 

As  discussed  in  Chapters  One  and  Two,  an  important  feature  of  PAT  is  its  ability  to  show  how 
cost-effectiveness  landscapes  vary  as  a  function  of  strategic  perspective.  A  perspective  is  rep¬ 
resented  by  the  different  sets  of  choices  used  in  scoring  and  aggregation.  Figure  3.5  shows  the 
Perspectives  sheet  for  our  continuing  example.  It  illustrates  the  format  for  entering  alternative 
perspectives,  starting  with  a  blank  sheet  (or  one  that  already  includes  a  default  perspective). 

Each  perspective  is  a  block  of  rows.  Each  block  has  a  header  row  with  the  name  of  the 
perspective;  the  rest  of  the  block  has  precisely  the  same  structure  as  the  top  portion  of  a  Level  2 
Data  sheet.  That  is,  it  includes  the  part  that  specifies  the  weighting  factors,  thresholds,  goals, 
and  so  on  for  each  Level  2  measure.  The  blocks  contain  no  information  about  the  options.  The 
blocks  corresponding  to  perspectives  are  juxtaposed  (i.e.,  there  is  no  space  between  them).  The 
weights  of  the  Level  2  measures  appear  in  the  header  line.  In  the  example,  the  weights  appear 
in  every  relevant  column,  but  it  is  permissible  to  have  them  appear  only  once  per  group. 

The  user  can  create  a  full  set  of  perspectives  directly  on  a  previously  blank  Perspectives 
sheet  by  adhering  to  the  syntax  of  the  example.  It  is  relatively  easy  to  make  data-entry  mistakes, 
however. 

Once  changes  are  made  in  a  given  perspective,  they  will  take  effect  only  if  one  of  them  is 
chosen  anew  in  the  Summary  sheet  as  the  current  perspective.  If  a  change  is  made  to  the  cur¬ 
rent  perspective  in  the  Perspectives  sheet,  the  user  should  select  a  different  current  perspective  in 
the  Summary  sheet’s  menu  and  then  change  again  the  perspective  in  question.  That  will  refresh 
the  current  perspective  appropriately. 

Currently,  perspectives  can  be  defined  only  with  respect  to  Level  2  data,  except  through 
the  method  of  extended  perspectives  discussed  below. 


Figure  3.5 
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110  1 

1  0 
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Easier  Ways  to  Create  and  Store  Perspectives 

In  practice,  the  easiest  way  to  create  perspectives  may  be  by  working  with  the  measure  weights 
and  the  Current  Perspective  menu  in  the  Summary  sheet.  The  procedure  (also  described  in 
Chapter  Four)  is  as  follows: 

•  In  the  Summary  sheet,  change  Level  1  measure  weights  to  those  appropriate  for  the  new 
perspective.  If  desired,  also  change  the  scoring  method  used  (e.g.,  Weakest  Link  instead 
of  Thresholds).  If  desired  (this  is  more  unusual),  go  to  the  relevant  Level  2  Data  sheet  (the 
normal  one  or  the  MRM  sheet)  and  change  any  of  the  control  parameters  that  control 
scoring  and  aggregation,  click  Modify  Summary,  and  then  return  to  the  Summary  sheet. 

•  Go  to  the  Current  Perspective  menu  (top  left)  and  select  Generate  New  Perspective. 

•  When  prompted,  fill  in  the  name  of  the  new  perspective  and  hit  return. 

•  PAT  will  automatically  copy  and  paste  appropriate  data  into  a  new  block  of  the  Perspec¬ 
tives  data  sheet. 

Figure  3.6  illustrates  the  Perspectives  menu  from  the  Summary  sheet.  It  indicates  the  pres¬ 
ence  of  four  perspectives  (Default,  Baseline,  Measure  1  Emphasis,  and  Weakest  Link),  just  as 
shown  in  Figure  3.5.  In  this  example,  we  included  the  weights  as  part  of  the  titles  so  that  we 
could  remember  what  they  represented. 

Some  other  rules  are  necessary  to  use  this  approach  effectively: 

1.  After  using  the  Current  Perspective  menu,  either  to  generate  a  new  perspective  or  to 
change  perspectives,  the  user  will  be  prompted  by  PAT  (Figure  3.7)  and  given  the 
opportunity  to  enter  a  perspective  name.  If  the  current  settings  (weights,  etc.,  have  been 
reset  for  a  new  perspective,  enter  that  name.  If  they  have  been  reset  to  change  the  defi¬ 
nition  of  an  existing  perspective,  type  the  old  perspective’s  name  and,  when  prompted 
again,  choose  Yes.  Otherwise  hit  OK,  without  entering  a  name.  This  query  box  appears 
routinely  when  closing  PAT,  even  if  no  changes  in  perspective  have  been  made.  The 
usual  response  is  OK. 

2.  If  errors  are  made  and  problems  arise,  corrections  to  existing  perspectives  can  usually 
be  made  by  editing  the  Perspectives  data  sheet.  In  some  cases,  however,  there  will  be  an 
error  message  about  inconsistencies  between  the  Level  2  Data  sheet  and  the  Perspectives 
sheet.  It  is  often  easiest  to  delete  everything  on  the  Perspectives  data  sheet  and  recreate 
the  perspectives  from  scratch.  That  avoids  tedious  trouble-shooting. 

Figure  3.6 

Using  the  Summary  Sheet's  Perspectives  Menu 
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Figure  3.7 

Query  About  Saving  or  Creating  a  New  Perspective 


Extended  Perspectives 

An  analyst  getting  into  the  spirit  of  exploring  the  consequences  of  alternative  perspectives  may 
wish  for  a  change  of  perspectives  that  would  have  more  consequences  than  are  allowed  by  the 
built-in  features  of  PAT,  which  apply  only  to  changes  at  Level  2  relating  to  weights,  goals, 
thresholds,  scoring  scheme,  and  aggregation  method. 

In  particular,  it  is  sometimes  logical  to  associate  a  perspective  with  changes  in  the  assess¬ 
ments  of  some  or  all  options,  not  just  the  weight  of  those  measures.  For  example,  a  conservative 
perspective  might  evaluate  risks  to  be  much  higher  than  an  optimistic  perspective  would.  Or, 
one  perspective  might  logically  weigh  Level  3  data  differently  than  another  perspective  would. 

Although  this  is  not  enabled  by  PAT’s  built-in  machinery,  extended  perspectives  can  be 
defined  by  someone  skilled  in  using  Excel.  This  might  involve,  for  example,  appending  some 
multipliers  to  the  data  elements  that  would  be  dependent  on  the  extended  perspective,  with  an 
interface  to  those  multipliers’  values  on  a  custom  sheet.  When  changing  an  extended  perspec¬ 
tive,  the  analyst  should  remember  to  change  the  multipliers’  values.  A  somewhat  safer  approach 
would  be  to  build  a  macro  that  would  be  used  each  time  PAT  ran  and,  depending  on  the 
extended  perspective  chosen,  would  set  the  parameter  values. 

Template  Builder 

We  have  mentioned  Template  Builder  repeatedly  throughout  this  chapter.  Figure  3.8  illustrates 
it  for  the  very  simple  example  used  in  most  of  this  report.  The  example  is  much  smaller  and  less 
busy  than  a  real  application  will  be.  The  inputs  are  in  red.  Although  it  is  conceptually  simple, 
some  rules  must  be  followed  to  fill  it  in  correctly: 

1.  Filling  in  the  Timeframe  (top  left)  is  simple,  as  is  picking  the  cost  units  from  the  menu 
item. 

2.  The  block  of  items  in  the  first  column,  Investment  Options,  must  have  the  options 
spelled  exactly  as  they  are  to  appear  on  displays.  This  list  is  unrelated  to  anything  else 
on  the  sheet.  That  is,  Option  A  has  nothing  to  do  with  Measure  1,  even  though  it  is 
adjacent  to  it. 

3.  The  Level  1,  Level  2,  and  Level  3  measures  are  entered  in  the  syntax  shown  in  the 
example,  which  means  that  their  relative  positioning  matters.  Thus,  Measure  1.1  and 
Measure  1.2  appear  in  cells  adjacent  to  the  cell  showing  Measure  1,  with  the  first 
of  them  being  right  beside  it.  Similarly,  for  the  one  example  of  a  Level  3  measure 
(Measure  2.2.1),  the  name  must  go  immediately  to  the  right  of  Measure  2.2.  One  of  the 
most  common  mistakes  in  using  PAT  is  getting  these  alignments  wrong. 
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Figure  3.8 

Template  Builder  for  a  Simple  Example 
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4.  The  Scoring  Method  and  High  or  Low  Values  Desired  columns  can  be  filled  in  straight¬ 
forwardly,  using  the  following  choices  (capitalization  matters): 

—  Scoring  Method:  Thresholds,  Weakest  Link,  Weak  Thresholds,  Rankings 

—  Goals 

—  High  or  Low  Values  Desired?:  High,  Low 

If  the  user  changes  these  settings  while  using  PAT,  by  going  into  the  Level  2  Data  or 
Level 3  Data  sheet,  for  example,  Template  Builders  data  will  be  rendered  obsolete.  That 
may  be  good  or  bad.  Template  Builder’s  data  could  be  regarded  as  a  default  to  which  one 
could  return,  or  one  might  be  worried  that  rerunning  Template  Builder  for  other  reasons 
would  introduce  errors.  Caution  is  necessary. 

5.  The  names  entered  under  Cost  Categories  have  nothing  to  do  with  the  rest  of  the  sheet. 

6.  The  names  entered  under  Investment  Items  have  nothing  to  do  with  the  rest  of  the 
sheet. 

7.  The  combination  of  Cost  Categories  and  Investment  Items  dictates  the  structure  of  the 
Cost  Data  sheet. 


The  Build  Sheets  command  causes  PAT  to  generate  structures  in  numerous  input  and 
output  sheets.  The  user  will  be  prompted  by  a  sequence  of  query  messages  that  ask  whether,  in 
rebuilding,  he  wishes  to  preserve  option-specific  data  already  in  the  Level  2  and  Level  3  Data 
sheets,  the  Perspectives  data  sheet,  and  the  Cost  Data  sheet.  In  many  cases,  saying  Yes  will  save 
a  great  deal  of  time  and  trouble  because  data  entry  is  time-consuming  and  prone  to  errors. 
Saying  Yes  is  appropriate  if,  for  example,  the  purpose  of  the  rebuilding  is  to  change  the  names 
of  some  measures  or  options,  or  to  add  some  additional  measures  or  options.  In  the  first  case 
(name-changing),  the  resulting  data  sheets  will  have  “holes”  wherever  newly  named  options  or 
measures  appear.  In  the  second  case  (adding  new  measures  or  options),  the  same  will  be  true. 
Obviously,  data  cannot  be  retained  if  they  do  not  exist. 
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Hint:  If  the  user  has  made  substantial  changes  in  the  data  structure,  or  if  errors  have 
appeared  that  troubleshooting  has  not  resolved,  it  may  be  better  to  rebuild  entirely  (saying 
No  rather  than  Yes  to  retaining  data),  even  though  this  will  mean  more  data  entry.  Errors 
often  are  the  result  of  subtle  misalignments  or  typos  (including  extra  spaces),  which  can 
be  erased  with  a  fresh  rebuilding.  After  the  new  sheets  are  generated,  the  user  may  use 
Copy  and  Paste  to  move  data  in  the  old  workbook  to  the  new  one,  being  careful  to  paste 
in  the  right  place.  A  common  error  is  pasting  into  what  appears  to  be  the  top  left  corner 
only  to  discover  that  scrolling  had  displaced  the  columns  so  that  the  pasting  was  actually 
onto  interior  columns. 


The  Show/Hide  button  can  be  used  to  toggle  on  or  off  some  in-sheet  examples  and 
instructions. 

After  Template  Builder  is  run,  the  various  input  and  output  sheets  will  have  a  good  deal 
of  structure  built  in.  This  does  not  include  option-specific  data  (unless  they  are  retained  from 
the  previous  version  of  PAT),  and  it  does  not  include  the  setting  of  control  variables  at  Levels  2 
and  3,  such  as  thresholds,  goals,  score  at  threshold,  and  score  at  goal.  It  also  does  not  include 
specifying  measure  weights  at  any  level.  In  some  cases,  PAT  will  build  in  default  values  (e.g., 
values  of  1  for  all  measure  weights  and  a  value  of  1  for  the  score  associated  with  raw  values  at 
or  exceeding  a  measure’s  goal).  However,  those  may  not  be  what  is  intended. 

The  bottom  line  here  is  that  even  with  Template  Builder,  data-entry  requirements  can 
be  considerable  and  should  be  undertaken  methodically  and  with  proofreading,  because  low- 
level  entry  errors  may  go  undetected.  In  the  vast  majority  of  cases  in  which  the  authors  have 
encountered  errors  in  PAT  over  the  past  year  or  two,  the  problems  originated  in  data  entry  and 
were  not  problems  with  PAT  itself.  A  few  additional  problems  had  to  do  with  misunderstand¬ 
ings.  Few  involved  bugs,  although  bugs  undoubtedly  remain — especially  for  the  least-exercised 
aspects  of  PAT  functionality. 
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Summary  Sheet 

Structure  of  the  Summary  Sheet 

The  Summary  sheet  is  the  main  output  of  PAT.  We  showed  its  schematic  form  in  Figure  2.7, 
but  Figure  4.1  shows  a  screenshot  of  the  actual  Summary  sheet.  Numerous  columns  have  been 
hidden  (see  gray  bars)  so  that  the  portions  could  be  juxtaposed.  Ordinarily,  users  work  with 
only  a  portion  of  the  sheet  at  a  time,  but  it  is  convenient  to  have  much  information  on  the  top- 
level  sheet. 

The  Summary  sheet’s  control  panels  provide  a  great  deal  of  flexibility.  All  employ  drop¬ 
down  menus,  the  first  of  which  is  the  Options  menu  (see  Figure  4.2). 

The  Options  menu  is  especially  important  for  setting  a  number  of  user  preferences.  Some 
items  are  self-evident;  some  are  less  so. 

1.  Update  Summary  Sheet.  The  user  should  update  the  displays  after  changing  data.  This  is 
the  command  to  select  when  the  user  is  in  the  Summary ;  some  equivalent  buttons  exist 
in  other  sheets. 

2.  Show/Hide  Weights.  Flaving  the  weights  of  the  various  Level  1  measures  shown  can 
be  toggled  on  or  off.  The  weights  are  used  in  the  calculation  of  effectiveness  and 
cost-effectiveness. 

3.  Change  Color  Scheme.  As  shown  in  Figure  4.3,  PAT  provides  four  different  color  schemes 
for  the  scoreboards:  the  standard  five-color  style,  a  style  with  letters  added  to  indicate 
the  colors,  a  style  with  alternative  colors,  and  a  gray-scale  style.  The  user  can  rotate 
through  the  set  of  styles  by  repetitively  selecting  Change  Color  Scheme.1  Adding  letters 
(top  right)  is  valuable  for  people  who  are  color-blind.  Gray  scale  is  sometimes  seen  as 
less  dramatic. 

4.  Update  Menus.  In  some  instances,  the  user  should  update  menus.  For  example,  if  the 
content  of  the  Perspectives  Cases  sheet  has  been  deleted  to  define  fresh  perspectives,  the 
menu  will  continue  to  show  the  old  ones  until  it  is  refreshed. 

5.  Generate  New  Perspective.  This  option  will  copy  current  settings  to  the  Perspectives  sheet 
with  whatever  name  is  specified  in  the  query  box  that  arises  after  the  option  is  chosen. 

6.  Update  Perspective.  This  option  replaces  data  in  the  Perspectives  sheet  with  data  being  used 
in  the  currently  operative  perspective  (whether  set  in  the  Summary  sheet  or  elsewhere). 


1  Further  tuning  of  colors  and  patterns  is  possible  using  Excel’s  built-in  capabilities,  but  doing  so  is  tedious  and,  in  our 
experience,  unrewarding,  because  of  Excel’s  limited  palette.  To  obtain  the  colors  intended,  it  may  be  necessary  to  recalibrate 
one’s  printer  or  computer. 
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Figure  4.1 

Illustrative  Summary  Sheet 
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Figure  4.2 

Options  Menu  of  Summary  Sheet 

Update  Summary  Sheet 
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Change  Color  Scheme 
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Generate  New  Perspective 
Update  Perspective 
Delete  Perspective 
Add/Remove  Failure  Markers 


Figure  4.3 

Alternative  Color  Schemes  for  Scorecards 
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7.  Delete  Perspective.  This  option  brings  up  a  query  box  in  which  the  user  can  specify  dele¬ 
tion  of  any  perspective  other  than  the  current  one. 

8.  Add/Remove  Failure  Markers.  This  is  an  option  to  show  (F)  in  cells  for  which  the  score 
falls  below  a  threshold. 

Continuing  with  the  other  control  panels  of  Figure  4.1,  sorting  is  controlled  by  two 
menus — one  specifying  the  criterion  for  sorting,  i.e.,  the  choice  of  column  (Figure  4.4)  and 
one  specifying  whether  the  sorting  will  be  in  ascending  or  descending  order  (Figure  4.5). 
Tire  choices  of  the  former  menu  change  with  usage  but  include  all  of  the  Summary  sheet  col¬ 
umns  (including  effectiveness  and  relative  cost-effectiveness),  the  various  cost  columns,  and 
any  Selected  Measure  columns  that  may  have  been  added  (the  columns  with  buttons  called 
Related  Details). 
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Figure  4.4 

Sorting-Categories  Menu  of  Summary 
Sheet  (Example-Specific) 
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Figure  4.5 

Sorting-Method  Menu  of  Summary  Sheet 


Largest  value  first 
Smallest  value  first 


As  discussed  in  Chapter  Three,  the  user  can,  at  any  given  time,  choose  to  work  with 
any  of  several  versions  of  his  data  sheet.  This  is  the  MRM  feature  for  which  the  choices  are 
(Figure  4.6)  entering  data  at  Levels  2  and  3  (referred  to  as  Use  Level  3  Data),  entering  data  at 
Level  2,  or  entering  data  at  Level  1. 

The  user  controls  the  scoring  method  (actually  the  method  of  aggregation),  which  some¬ 
times  has  substantial  effects.  The  choices  available  from  the  related  menu  (Figure  4.7)  are  those 
discussed  in  Chapter  Three.  The  Thresholds  method  is  the  default  choice.  If  no  change  is  made 
to  default  thresholds  and  goals,  the  familiar  method  of  linear  weighted  sums  is  used. 

Figure  4.6 

MRM  Menu  of  Summary  Sheet 

Use  Level  3  Data 
Use  MRM  Level  2  Data 
Use  MRM  Level  1  Data 


Figure  4.7 

Scoring-Method  Menu  of  Summary  Sheet 
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Current  perspective  can  be  changed  at  any  time,  generating  a  recalculation  of  results,  by 
using  the  corresponding  menu  (Figure  4.8).  The  entries  in  the  menu  are  based  on  the  perspec¬ 
tives  the  user  has  defined  so  far.  The  names  in  Figure  4.8  illustrate  using  shorthand  names  to 
remind  the  user  of  what  a  particular  perspective  means.  In  Baseline,  for  example,  the  weights 
of  the  three  measures  of  our  continuing  example  are  all  1,  whereas  they  are  different  in  the  next 
two  perspectives.  The  last  perspective  changes  both  the  emphasis  (via  weights)  and  the  scoring/ 
aggregation  scheme. 

The  final  control  menu  (Figure  4.9)  allows  the  user  to  specify  the  discount  rate,  which  is 
assumed  to  be  between  0  and  0.1,  corresponding  to  0  to  10  percent.  If  Cost  Data  sheet  items  are 
in  constant  (inflation-adjusted)  dollars,  this  menu  item  should  be  seen  as  the  “real”  discount 
rate,  the  earning  power  of  money  above  the  inflation  rate.  If  Cost  Data  sheet  items  are  in  cur¬ 
rent  dollars,  the  menu  item  should  be  seen  as  the  sum  of  the  inflation  rate  and  the  discount  rate 
to  be  assumed  in  Summary  sheet  calculations. 

We  recommend  that  users  specify  the  basic  structure  of  their  sheets  by  using  Template 
Builder,  as  discussed  at  the  end  of  Chapter  Three.  Most  of  the  Summary  sheet’s  columns  will 
then  be  generated  with  appropriate  headers  and  values  when  PAT  is  first  run  (by  selecting 
Update  Summary  from  the  Options  menu).  If  the  user  wishes  to  add  or  delete  columns  or 
options,  it  is  usually  best  to  do  that  in  Template  Builder  and  rebuild  the  sheets.  However, 
changes  can  also  be  made  as  follows  (with  the  disadvantage  that  they  will  not  be  reflected  in 
the  Template  Builder  sheet). 


Figure  4.8 

Current-Perspective  Menu  of  Summary 
Sheet  (Example-Specific) 


Default 
Baseline  (1,1,1) 

Measure  1  Emphasis  (2, 1,0. 5) 
Weakest  Link  (2,1, 0.5) 


Figure  4.9 

Discount-Rate  Menu  (Summary  Sheet) 
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Structuring  Rows  and  Columns  with  Template  Builder 

Looking  back  to  Figure  4.1  and  ignoring  the  control  panels  at  the  top  and  the  mapping  of 
colors  at  the  bottom,  from  left  to  right,  for  each  investment  option  there  are  four  blocks  of 
output  data,  the  dimensions  of  which  are  hard-wired  in  PAT: 

•  Up  to  12  columns  in  a  scorecard  showing  results  for  Level  1  measures. 

•  Up  to  four  columns  displaying  numeric  data  (e.g.,  for  selected  measures  from  Level  2). 

•  Up  to  nine  columns  displaying  cost  data. 

•  Up  to  two  additional  columns  presenting  the  effectiveness  and  relative  cost-effectiveness 
of  the  investment  options. 


Adding  or  Deleting  a  Column 

The  user  can  add  any  Level  1  measure  that  has  been  introduced  in  the  data  sheets  to  the  Sum¬ 
mary  scorecard  simply  by  typing  the  measure’s  precise  name  in  Row  1  for  any  of  the  columns 
set  aside  for  the  scorecard — nominally,  columns  B  through  M  (the  scorecard’s  “range”),  but  in 
any  case,  columns  with  Detail  buttons.  To  delete  any  column  from  the  scorecard,  the  user  may 
erase  the  name  in  the  header  and  select  Update  Summary  from  the  Options  menu. 

Adding  a  Numeric  Column  of  Level  2  Information 

The  user  may  wish  to  elevate  the  visibility  of  a  particular  Level  2  measure  by  including  its 
numerical  values  in  the  Summary  sheet,  using  the  next  block  of  columns  indicated  in 
Figure  4.1.  These  are  nominally  Columns  O  to  R,  the  columns  with  Related  Details  buttons. 
The  procedure  is  as  follows:  In  Row  1,  in  any  column  in  the  reserved  range,  type  the  name  of 
the  Level  1  measure,  followed  by  the  name  of  the  particular  Level  2  measure,  separated  by  :: 
(with  no  spaces).  An  example  would  be  Measure  2::Measure  2.1.  Copying  and  pasting  a  name 
from  the  Level  2  Data  sheet  may  help  avoid  typographical  errors. 

Altering  Cost-Related  Columns 

The  columns  set  aside  for  cost  data  (nominally  T  through  AB,  those  with  Cost  Detail  buttons) 
can  be  used  to  display  subsets  of  information  in  the  Cost  Data  sheet.  Take  a  case  in  which  PAT 
was  set  up  initially  to  display  R&D,  acquisition,  and  O&S  costs  on  the  Summary  sheet  for  the 
period  2010-2015.  PAT  will  add  a  Total  Cost  column  for  that  period  as  well.  But  then  suppose 
the  user  wants  to  show  two-year  costs  as  well.  The  additional  columns  could  be  added  to  the 
summary  by  simply  typing  in  the  correct  names  in  Row  1,  probably  in  columns  X,  Y,  and  Z 
(or  at  least  in  columns  with  Cost  Detail  buttons).  In  this  case,  the  user  would  type  in  the  fol¬ 
lowing  names,  respectively:2 

R&D  Cost:  2010-2011  ($M) 

Acquisition  Cost:  2010-2011  ($M) 

O&S  Cost:  2010-2011  ($M) 

To  specify  a  single  year,  one  should  use  just  that  year  rather  than  a  range  (i.e.,  2010,  rather  than 
2010-2010). 


2  The  ($M)  is  not  included  in  the  cost-category  name,  but  is  added  separately.  The  names  being  added  cannot  be  simply 
copied  and  pasted  from  the  Cost  Data  sheet. 
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Figure  4.10  illustrates  this  example,  showing  the  cost  columns  of  the  Summary  sheet.  The 
columns  for  the  2010-2015  interval  would  be  generated  automatically,  but  the  column  for  the 
period  2010-2011  has  been  added,  as  mentioned  above,  by  typing  in  Total  Cost  (2010-2011) 
($M).  PAT  understands  the  word  “Total”  to  mean  the  sum  of  the  costs  over  the  specified  cost 
categories,  which  in  this  case  are  R&D,  acquisition,  and  O&S. 

To  show  all  categories  of  costs  for  two  years  rather  than  the  six  years,  the  user  could 
merely  edit  the  column  headers,  changing  2010-2015  to  2010-2011. 

Adding  or  Deleting  Options 

Analysts  using  PAT  will  frequently  enter  data  for  more  options  than  can  reasonably  be  dis¬ 
played.  However,  minds  change  and  it  is  often  necessary  to  add  or  delete  an  option  in  the  Sum¬ 
mary  display.  This  is  done  simply  by  inserting  a  row  within  the  scorecard  range  and  typing  in 
the  precise  name  of  the  option  in  Column  A. 

Cost-Effectiveness 

The  two  rightmost  columns  in  Figure  4.1  are  labeled  by  PAT  as  Effectiveness  and  Cost- 
Effectiveness.  The  method  used  to  calculate  effectiveness  is  selected  from  the  Cost  Effective¬ 
ness  Scoring  Method  dropdown  menu;  the  cost  metric  used  in  the  cost-effectiveness  cal¬ 
culation  is  selected  from  the  Cost-Effectiveness  Cost  Metric  dropdown  menu.3  To  make 
present-value  calculations,  the  user  can  apply  a  discount  rate  to  the  cost  numbers,  using  the 
Discount  Rate  dropdown  menu.  This  should  be  understood  as  the  real  discount  rate  if  the 
costs  in  the  Cost  Data  sheet  are  already  corrected  for  inflation,  and  as  the  sum  of  inflation  and 
real  inflation  rate  otherwise. 

The  cost-effectiveness  values  are  scaled  so  that  the  most  cost-effective  investment  option 
has  a  value  of  1,  and  all  other  investment  options  are  compared  to  it.  A  no-changes  baseline,  if 
present,  will  always  have  a  cost-effectiveness  of  0  (rather  than  allowing  it  to  be  infinity  because 
it  involves  zero  cost). 


Figure  4.10 

Cost  Information  in  the  Summary  Sheet 
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3  This  menu  sometimes  has  duplicate  versions  of  the  same  costs,  with  slight  differences  in  names. 
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Comments,  Flags,  and  Warnings 

The  cell  showing  the  name  of  an  investment  option  may  have  a  comment,  indicated  by  a  red 
triangle  in  the  upper  right  corner.  Mousing  over  that  cell  brings  up  the  comment,  which  might 
be  a  cryptic  definition  or  nuance  (Figure  4.11). 

If  a  cell  in  the  scorecard  has  a  similar  flag,  the  flag  is  a  warning.  Mousing  over  it  will  cause 
a  dropdown  to  appear  with  a  brief  description  of  what  is  being  warned  about  (see  Figure  4.11). 

Finally,  mousing  over  the  name  of  a  measure  brings  up  any  comment  (from  the  Level  1 
Data  sheet)  that  may  describe  the  measure;  it  will  also  show  the  measure’s  weight  in  effective¬ 
ness  calculations. 

Measure  Weights 

The  weights  of  the  various  Level  1  measures  used  in  calculating  effectiveness  can  be  shown  or 
hidden  as  a  group  by  selecting  Show/Hide  Weights  from  the  Options  menu.  If  the  weights  are 
shown  (just  below  the  Detail  buttons),  they  can  be  changed  directly,  in  which  case  PAT  re¬ 
calculates  effectiveness  and  relative  cost-effectiveness  immediately,  without  prompting.  PAT 
also  modifies  the  weights  of  the  measures  on  the  Level  1  Data  sheet. 

Buttons 

The  Summary  sheet  includes  various  buttons.  Clicking  a  Detail  button  will  bring  up  the 
Level 2  Drilldown  sheet  for  the  particular  measure  (column).  Clicking  a  Related  Details  button 
for  any  of  the  numeric  data  columns  will  also  bring  up  the  relevant  Level  2  Drilldown  sheet. 
Clicking  the  Cost  Detail  button  opens  the  Cost  Charts  sheet.  There  is  no  button  directing 
access  to  Level  3  information.  One  accesses  Level  3  Drilldown  as  discussed  in  the  subsection 
on  that  topic  below. 

An  Illustrative  Summary-Level  Scorecard 

Figure  4.12  shows  a  condensed  version  of  the  scorecard  portion  of  the  Summary  sheet  for  our 
simple  example.  We  have  chosen  to  display  only  the  answers  plus  the  measure  weights,  ignor¬ 
ing  the  other  features  of  the  scorecard.  The  red  rectangle  indicates  that  we  will  drill  down  for 
more  information  on  Measure  2. 

Figure  4.11 

Warning  Flags  in  a  Summary  Sheet 
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Figure  4.12 

Illustrative  Summary  Scorecard  (Level  1) 


Level  2  Drilldown  Sheet 

Clicking  on  the  Detail  button  for  any  particular  measure  in  the  Summary  scorecard  brings 
up  the  Level  2  Drilldown  sheet,  as  Figure  4.13  illustrates  for  Measure  2.  This  sheet  includes  a 
scorecard  showing  the  scores  of  all  contributors  to  the  measure’s  effectiveness  and,  on  the  right 
side,  the  calculated  consequences  (which  are  the  same  as  the  column  of  scores  or  colors  shown 
in  the  Level  1  scorecard  for  Measure  2,  reading  downward:  green,  yellow,  orange).  In  more 
typical  cases,  the  scorecard  may  have  from  three  to  seven  Level  2  measures  and  will  therefore 
be  more  complex. 

The  default  is  that  Level  2  drilldown  is  shown  for  the  same  settings  as  specified  in  the 
Summary  (e.g.,  Thresholds).  However,  the  user  can  vary  the  scoring  method  and  weights  in 
this  sheet  in  order  to  test  some  variations.  He  should  use  the  buttons  at  the  top  right  to  make 
the  changes  take  effect,  and  he  should  be  cautious  about  using  different  scoring  systems  in  the 
Summary  sheet  and  this  sheet. 

Depending  on  which  options  are  selected  from  the  two  Display  menus,  only  some  of  the 
information  in  the  sheet  is  shown.  Users  will  not  always  want  to  see  the  information  above  the 
scorecard.  For  production  purposes,  the  color-explanation  chart  at  the  very  bottom  is  also  not 
very  interesting.  Figure  4.14  shows  a  screenshot  with  many  of  the  upper  rows  suppressed  (using 
the  Display  menu)  and  with  the  color-explanation  chart  left  out.  The  rectangle  indicates  that 
we  will  drill  down  on  Measure  2.2.  The  italic  letters  in  the  boxed  column  indicate  that  this 
measure  is  calculated  from  Level  3. 

At  the  price  of  disrupting  flow  from  Figure  4.14  to  the  Level  3  Drilldown  sheet,  let  us  first 
note  that  Level  2  Drilldown  displays  look  somewhat  different  when  other  scoring/aggregation 
methods  are  used.  If  the  Goals  scoring  method  is  selected,  cells  will  be  green  or  red,  depend¬ 
ing  on  whether  or  not  the  raw  value  for  each  Level  2  measure  reaches  the  goal  (Figure  4.15). 
If  it  does  so,  the  rightmost  column  gives  the  weighted  percentage  of  goals  achieved  over  all  of 
the  Level  2  measures  (for  the  particular  Level  1  measure).  The  color  of  the  cell  in  the  right¬ 
most  column  corresponds  to  the  color  of  the  corresponding  cell  on  the  Summary  sheet.  If  the 
Thresholds,  Weak  Thresholds,  or  Weakest  Link  scoring/aggregation  method  is  selected,  cells 
in  the  table  are  colored  red,  yellow,  or  green,  depending  on  whether  the  raw  value  (1)  does  not 
reach  the  threshold  value,  (2)  reaches  the  threshold  value  but  does  not  reach  the  goal  value,  or 
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Figure  4.13 

Level  2  Drilldown  Sheet 
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(3)  reaches  the  goal  value,  respectively.  The  color  of  the  cell  corresponds  to  the  measure  score, 
with  the  maximum  score  being  1  (corresponding  to  green)  and  the  minimum  score  being  zero 
(corresponding  to  red). 

For  the  Rankings  scoring  method  (Figure  4.16),  each  Level  2  measure  is  ranked  individu¬ 
ally,  with  colors  on  a  light  blue  to  dark  blue  scale  showing  how  the  investment  options  rank 
relative  to  one  another.  The  rightmost  column  shows  a  weighted  average  ranking.  The  score 
and  color  of  the  cell  will  be  carried  up  to  the  Summary  sheet  for  that  investment  option  and 


measure. 
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Figure  4.14 

Compressed  Version  of  Level  2  Drilldown  for  Measure  2 
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Figure  4.15 

Level  2  Drilldown  with  Goals  Method 


Figure  4.16 

Level  2  Drilldown  with  Rankings  Method 
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The  raw  values  and  the  scoring-function  parameters  may  be  edited  in  the  Drilldown 
sheets,  even  though  they  are  nominally  output  sheets.  Once  the  data  and  parameters  have  been 
changed,  the  user  should  click  on  the  Update  Level  2  Data  button,  which  will  update  the  data 
sheets  and  will  also  recreate  the  Summary  and  Drilldown  sheets  with  the  new  information. 


Level  3  Drilldown  Sheet 

Returning  now  to  the  stream  of  examples  using  the  Thresholds  scoring  method,  and  pick¬ 
ing  up  with  the  drilldown  shown  in  Figure  4.13,  the  Level  3  Drilldown  sheet  is  shown  in 
Figure  4.17.  To  access  it  requires  a  small  change  of  procedure.  The  user  must  first  click  on  the 
Level  3  Drilldown  tab  or  go  to  that  sheet  using  the  Go  To  Sheet  menu  at  the  top  of  the  Excel 
window.  When  the  next  screen  comes  up,  the  user  must  take  the  additional  step  of  selecting 
the  desired  Level  3  measure  from  the  Level  3  Drilldown  Options  menu  at  the  top  left  (not 
shown).  The  result  is  shown  in  Figure  4.17  for  our  continuing  simple  example.  In  this  chart, 
however,  we  have  chosen  (using  the  display  menus)  to  show  the  numerical  raw-value  figures 
that  determine  the  colors.  They  might  be  suppressed  in  a  presentation,  but  the  analyst  some¬ 
times  finds  them  useful. 

Note  that  the  final  column  shows  scores  of  green,  yellow,  and  red  (reading  downward), 
which  agrees  with  the  column  for  Measure  2.2  in  Figure  4.13. 

Figure  4.17 

Level  3  Drilldown  for  Measure  2.2 
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The  reader  may  wonder  why  such  drilldown  charts  have  two  apparently  identical  columns 
on  the  right,  entitled  Measure  2.2  Score  and  Level  2  Score.  To  explain,  suppose  that  we  change 
the  assumptions  of  our  simple  problem  slightly,  by  specifying  a  threshold  value  for  Level  2 
Measure  2.2.  In  that  case,  results  would  change,  and  the  Level  3  explanation  would  look  like 
Figure  4.18. 

If  we  scan  the  colors  at  the  left  of  the  row  for  Option  A  (ignoring  the  numbers  for  a 
moment),  it  may  seem  that  they  don’t  average  correctly.  For  example,  Measures  2.2.1  and  2.2.2 
are  light  green  and  green,  averaging  to  green  in  the  Measure  2.2  Score  column.  That  is  per¬ 
haps  reasonable.  But  why,  then,  is  the  Level  2  score  yellow?  We  will  walk  through  the  calcula¬ 
tions,  because  they  are  somewhat  subtle.  Note  first  from  the  rows  above  the  scorecard  that  the 
Measures  2.2.1  and  2.2.2  have  Level  3  thresholds  of  5  and  5.  Thus,  averaging  the  raw  values 
in  this  case  produces  (1/2)  [(8  -  5)15  +  (10  -  5)/5]  or  0.8,  as  shown  in  the  Measure  2.2  Score 
column.  That  is,  the  scoring  “starts”  at  a  raw  value  of  5.  In  this  case,  however,  Level  2  also  has 
a  threshold.  The  small-print  parenthetical  expression  below  the  Level  2  score  tells  us  about 
this  without  our  having  to  look  in  the  Level  2  Drilldown  sheet.  The  syntax  of  (0.51,  0,  1,  1) 
means  (see  the  note  at  the  bottom  of  the  display)  that  the  threshold  is  0.51,  the  score  at  the 

Figure  4.18 

Level  3  Drilldown  If  Measure  2.2  Has  a  Threshold  Value 
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threshold  is  0,  the  goal  is  1,  and  the  score  at  the  goal  is  1.  The  value  of  0.8  to  be  evaluated  (the 
Measure  2.2  score)  is  59  percent  of  the  way  between  0.51  and  1.  Thus,  the  Level  2  score  is  0.59, 
as  shown,  which  is  just  within  the  range  associated  with  yellow. 

PAT’s  flexibility  in  allowing  the  user  to  attach  goals  and  thresholds  is  powerful,  but  using 
goals  and  thresholds  requires  caution  and  attention  to  detail — especially  if  they  are  used  at 
both  Levels  2  and  3. 4  One  price  is  that  a  “visual  explanation”  may  not  be  enough  in  drilldowns; 
it  may  be  necessary  to  look  at  the  details,  as  with  Figure  4.13. 


Cost  Data  Sheet 

Costs  are  an  input  to  PAT,  as  discussed  in  Chapter  Three  (see  Figure  3.4),  but  PAT  also  gener¬ 
ates  output  charts.  If,  for  example,  the  user  specifies  cost  streams  for  categories  R&D,  acquisi¬ 
tion,  and  O&S  by  option,  by  year,  the  Cost  Data  sheet  will  show  plots  of  yearly  or  cumulative 
cost,  by  option  and  category,  or  total  costs  across  categories.  Figure  4.19  illustrates  the  latter. 
It  expresses  costs  in  current  dollars  (also  called  current-year  or  then-year  dollars),  but  the  user 
will  often  wish  to  use  constant  dollars  (also  called  real  dollars)  or  present-value  calculations,  as 
discussed  in  Chapter  Three. 

Figure  4.19 

Illustrative  Total  Costs  Versus  Time  Chart 


Year 
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—  Option  B 
Option  C 


NOTE:  The  example  uses  current  (i.e.,  then-year)  dollars,  but  the  user  can  specify  use  of  constant  dollars 
instead. 


4  An  example  of  this  complexity  might  be  a  set  of  Level  3  measures  related  to  risk. 
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Scatter  Plot  Sheet  for  Cost-Effectiveness  Landscapes 

PAT’s  built-in  Scatter  Plot  sheet  allows  the  user  to  construct  cost-effectiveness  landscapes,  i.e., 
plots  of  effectiveness  (or  another  measure  of  performance)  versus  cost.  The  user  can  choose  any 
one  cost  metric  and  any  two  evaluation  metrics  from  menus  on  the  Scatter  Plot  sheet.  The  cost 
metrics  on  the  menu  are  generated  from  the  cost  categories  and  any  specialized  cost  metrics 
used  on  the  Summary  sheet.  The  evaluation  metrics  may  be  any  of  the  Level  2  measures  or  an 
effectiveness  score  calculated  by  any  of  the  built-in  methods  (e.g.,  Threshold  or  Weakest  Link). 
The  chart  presents  points  representing  the  options  located  by  cost  (x-axis)  and  effectiveness 
metric  (vertical  y-axis  or  axes).  Mousing  over  a  point  reveals  which  option  the  point  refers  to. 

These  cost-effectiveness  landscapes  are  quite  useful  when  evaluating  choices  with  eco¬ 
nomic  constraints.  They  can  illustrate  both  the  classic  phenomenon  of  diminishing  returns 
and,  e.g.,  the  “chunkiness”  phenomenon,  in  which  as  cost  increases,  larger  increases  are  neces¬ 
sary  before  there  are  significant  increases  in  effectiveness. 

Figure  4.20  shows  a  scatter  plot  with  six  options.  In  this  case,  the  behavior  is  classic: 
Effectiveness  rises  with  cost,  but  at  a  diminishing  rate.  And  at  any  given  cost  level,  there  are 
better  and  poorer  performers. 

Figure  4.21  shows  the  Scatter  Plot  sheet  for  the  simple  example  used  in  this  report,  which 
has  only  three  options.  It  also  shows  the  controls  for  the  scatter  plot.  For  this  display,  we  chose 
to  evaluate  the  options  for  both  the  Thresholds  and  Weakest  Link  methods.  In  this  case,  the 
relative  goodness  of  the  options  is  unchanged,  but  that  will  not  always  be  so. 

Figure  4.20 

Illustrative  Scatter  Plot 
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NOTE:  The  points  are  individual  options.  Their  names  are  revealed  by  mousing  over 
them  or  by  changing  the  relevant  chart  option.  Costs  in  this  figure  are  given  in 
current  (then-year)  dollars. 
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Figure  4.21 

Scatter  Plot  for  the  Simple  Problem,  Using  Two  Scoring  Methods 


Changing  the  evaluation  metric  or  any  of  the  investment  options  will  automatically 
regenerate  the  Scatter  Plot,  as  will  clicking  on  the  Generate  Scatterplot  button,  which  should 
be  used  when  changes  are  made  in  the  data  for  the  various  options  given  in  the  Levels  2  and  3 
databases.  The  scatter  plot  will  not  update  for  such  changes  automatically,  in  part  because  it  is 
convenient  to  have  the  chart  remain  constant  while  one  looks  at  other  sheets.  Another  reason 
for  constancy  is  that  when  Excel  regenerates  charts,  it  restores  default  formatting,  which  may 
undo  a  good  deal  of  work.  If  the  user  does  want  automatic  regeneration,  however,  he  needs 
only  to  click  the  appropriate  box. 


Spider  Charts  Sheet 

The  Spider  Charts 5  sheet  (Figure  4.22)  allows  the  user  to  select  a  measure  and  up  to  four 
investment  options.  The  goal  and  threshold  values  for  the  Level  2  measures  may  be  inputted 
instead  of  two  of  the  investment  options.  The  Level  2  measures  of  the  selected  Level  3  mea¬ 
sure  are  shown  as  arms  of  the  spider  chart.  The  values  in  the  spider  chart  are  scaled  relative  to 
the  investment  option  (or  goal  or  threshold  value)  selected  as  Chart  1.  Also,  PAT  inverts  the 
scales,  if  necessary,  so  that  in  all  cases,  more  is  better;  i.e.,  the  farther  out  an  option  extends, 
the  better.  There  are  no  values  on  the  axes  because  the  purpose  of  the  chart  is  merely  to  com¬ 
municate  visually  a  sense  of  relative  effectiveness  by  different  measures.  Changing  the  measure 
to  be  charted  or  any  of  the  investment  options  will  automatically  regenerate  the  spider  chart,  as 


^  Spider  charts  are  also  called  “radar  charts.’ 
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Figure  4.22 

Illustrative  Spider  Plot 
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will  clicking  on  the  Generate  Spider  Chart  button,  which  should  be  used  when  the  raw  values 
or  parameters  on  the  data  sheets  are  changed. 


Multimeasure  Spider  Charts  Sheet 

Multimeasure  spider  charts  (Figure  4.23)  provide  a  visual  depiction  across  different  Level  1 
measures.  For  each  investment  option  displayed  on  the  spider  chart,  the  values  along  each  arm 
correspond  to  the  average  rank  for  the  Rankings  scoring  method  or  the  measure  score  for  the 
other  scoring  methods.  For  the  Thresholds  scoring  method,  if  any  investment  option  for  a  sub  ¬ 
measure  fails,  the  value  for  the  corresponding  measure  falls  inside  a  red  polygon  representing 
failed  measures.  The  user  can  select  the  scoring  method,  using  the  Scoring  Method  dropdown 
menu,  and  up  to  four  investment  options.  Changing  either  will  regenerate  the  spider  chart,  as 
will  clicking  on  the  Generate  Spider  Chart  button,  which  should  be  used  when  the  raw  values 
or  parameters  on  the  data  sheet  are  changed. 
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Figure  4.23 

Multimeasure  Spider  Plot 
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Selected  Details  Sheet 

Figure  4.24  generates  a  Selected  Details  display  similar  to  the  Drilldown  sheets,  except  that 
the  Level  2  measures  in  the  columns  are  specifically  chosen  for  the  purposes  of  the  particular 
analysis  for  which  PAT  is  being  used  and  may  come  from  different  measures.  That  is,  whereas 
the  Drilldown  sheets  relate  to  a  particular  measure,  the  Selected  Details  sheet  allows  the  user  to 
tailor  a  display  with  details  selected  from  several  measures.  As  with  the  data  sheets,  all  scoring 
methods  and  the  associated  color  schemes  may  be  displayed  on  the  Selected  Details  sheet  by 
using  the  Scoring  Method  dropdown  menu.  The  dropdown  menus  allow  selection  of  the  avail¬ 
able  measures,  using  the  format  Measure::Submeasure. 


Rankings  Table  Sheet 

Figure  4.25  shows  the  Rankings  Table  sheet,  in  which  the  investment  options  are  ranked  by 
effectiveness  (or  cost-effectiveness)  for  each  perspective  defined  on  the  Perspectives  sheet.  The 
scoring  method  and  cost  metrics  used  can  be  selected  from  menus.  In  each  cell,  the  rank  is 
shown  in  large  bold  text,  and  the  relative  (cost-) effectiveness  appears  in  parentheses  under  the 
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Figure  4.24 

Selected  Details  Sheet 
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Figure  4.25 

Rankings  Table  Sheet  for  Effectiveness  or  Relative  Cost-Effectiveness 
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ranking.  The  investment  option  with  the  highest  absolute  (cost-) effectiveness  for  each  perspec¬ 
tive  is  given  a  relative  cost-effectiveness  of  1,  with  all  other  cost-effectiveness  values  measured 
against  it.  The  same  scaling  is  used  with  effectiveness  (not  shown). 


CHAPTER  FIVE 


Details  of  the  Methodology 


This  chapter  repeats  some  of  the  earlier  material  from  Chapter  Two  in  order  to  be  relatively 
self-contained,  but  its  purpose  is  to  generalize,  lay  out  the  mathematics,  comment  on  some  of 
the  subtleties  of  the  methodology,  and  work  through  an  example. 


Basic  Concepts  and  Definitions 

Figure  5.1  indicates  how  PAT  operates  analytically  (except  with  the  Rankings  method  discussed 
below,  under  Alternative  Methods).  For  each  investment  option,  PAT  takes  a  series  of  inputs 
(grayed  items)  and  characterizes  how  well  the  option  performs  by  different  criteria,  includ¬ 
ing  cost-effectiveness.  The  illustrative  figure  assumes  n  measures,  of  which  only  the  first  and 
the  wth  are  shown.  A  given  measure  might  be  calculated  from  a  combination  of  Level  2  and 
Level  3  data  (as  shown  for  Measure  1)  or  might  be  specified  directly  (as  shown  for  Measure  n).' 

The  following  paragraphs  explain  the  PAT  concepts  and  terminology. 

Attributes  of  Investment  Options 

The  investment  options  are  alternative  programs  of  investment  over  time.  Each  investment 
option  has  a  number  of  attributes,  described  below. 

Input  Costs.  Each  PAT  investment  option  must  have  an  associated  investment  stream, 
specified  by  projected  cost  for  each  year,  cost  category  (e.g.,  R&D,  acquisition,  and  O&S), 
and  expenditure  item  (e.g.,  ships,  aircraft)  of  interest  for  the  analysis.  Such  data  can  be  rather 
complex  but  can  be  simplified  to,  e.g.,  by  using  a  single  category  (Total),  a  single  type  of  item 
(General),  or  a  short  time  period. 

The  required  cost  streams  are  inputs  expressed  in  current  (i.e.,  then-year)  dollars  or  dollars 
corrected  for  inflation.  The  initial  data  available  to  the  PAT  analyst  will  often  be  a  combination 
of  detailed  cost  streams  and  fragments  of  other,  more  aggregated  information  such  as  the  pro¬ 
jected  20-year  or  Future-Years  Defense  Plan  (FYDP)  cost  of  a  weapon  system  or  type  of  force 
unit.  The  PAT  user,  then,  must  draw  on  such  heterogeneous  information  to  estimate  the  cost 
streams  required  for  PAT  as  best  he  can.  He  may  choose  to  make  simplifying  approximations, 
such  as  spreading  aggregate  costs  equally  over  the  entire  time  period. 


1  As  a  matter  of  implementation,  to  specify  a  Level  1  measure  directly  in  PAT,  one  defines  the  measure  as  a  function  of  only 
one  Level  2  measure.  It  is  that  Level  2  measure  that  is  directly  inputted. 
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Figure  5.1 

Schematic  of  PAT's  Calculations 


- ►  1 :1  mapping 

- ►  n:1  mapping  (aggregation) 

F:  function 

NOTE:  Grayed  items  are  inputs. 

Output  Cost.  After  constructing  PAT-consistent  cost  data  (typically  in  inflation-adjusted 
dollars)  for  his  options,  the  analyst  may  use  PAT  (or  custom  worksheets  within  PAT)  to  gener¬ 
ate  cost  information  in  many  different  forms.  Some  of  this  activity  will  require  merely  manip¬ 
ulating  the  core  data  in  different  ways,  but  some  will  involve  calculation.  For  example,  the 
analyst  may  wish  to  generate  present-value  costs  for  various  options.2  If  so,  he  must  specify  a 
real  discount  rate  (or  a  set  of  rates  to  use  for  bounding  the  problem).  The  formula  used  for  the 
present  value  of  a  set  of  payments  to  be  received  over  n  years  is 


i=l 


(1  +  DR)‘ 


2  A  promise  to  pay  someone  $1  million  in  ten  years  is  less  painful  than  paying  $1  million  now,  because  one  could  invest  the 
$1  million  for  ten  years,  pay  the  debt  at  that  point,  and  keep  the  returns.  The  economic  calculations  are  well  understood,  as 
discussed  in  many  sources,  including  Wikipedia  (Hitch  and  McKean,  1965),  but  the  real  discount  rate  is  controversial  and 
dependent  on  the  particular  problem  considered.  It  is  arguably  good  practice  to  calculate  for  real  discount  rates  of  both  0.03 
and  0.07  to  bound  the  calculations.  For  discussion  of  how  the  Government  Accountability  Office  (GAO)  came  to  suggest 
this,  see  Graham,  2007.  Such  issues  were  addressed  in  a  recent  RAND  study  of  the  resource  implications  of  alternative  U.S. 
global  military  strategies  (Davis  et  ah,  2008). 
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where 

DR  =  real  discount  rate 

Ei  =  payment  promised  at  end  of  zth  year,  in  real  (inflation-protected)  dollars. 

The  sign  is  positive  or  negative,  depending  on  whether  one  is  receiving  or  paying  and  on  the 
syntax  of  discussion.  If  the  yearly  expenditures  are  expressed  in  current  (i.e.,  then-year)  dollars 
(without  correction  for  inflation),  the  formula  is 


pv=j^ 

i= 1 


(i+/)'(i+i>y 


tt(i+DRy 


where 

D  =  nominal  discount  rate 

E-  =  payment  promised  at  end  of  zth  year,  in  real  (inflation-protected)  dollars. 

DR  =  D  +  I 

The  approximation  is  good  except  for  very  high  rates  of  inflation.  From  a  simple  Taylor’s 
expansion,  the  first-order  correction  would  be  a  factor  of  /,  i.e.,  0.97  for  3-percent  inflation. 

Measures  and  Submeasures  (Level  1,  2,  and  3  Measures),  Raw  Values,  and  Scores 

The  options  are  characterized  at  the  Summary  level  by  criteria  called  “measures”  or  “Level  1 
measures,”  each  of  which  has  one  or  more  submeasures  at  Levels  2  or  3,  or  both.  We  refer  to  the 
measures  in  shorthand  as  Level  1,  Level  2,  and  Level  3  measures.  If  we  refer  to  submeasures, 
we  include  both  Level  2  and  Level  3  measures  or  just  Level  3  measures,  depending  on  context. 
A  given  input  to  PAT  is  usually  made  at  either  Level  2  or  Level  3.  If  a  particular  input  is  made 
at  Level  3,  then  Level  2  information  is  calculated.  As  is  the  case  throughout  mathematics  and 
computer  science,  the  same  label  (e.g.,  “a  missile’s  single-shot  kill  probability”)  is  used  to  refer 
to  the  abstract  concept  that  a  measure  or  submeasure  represents  and  also  the  value  ascribed  to 
that  measure. 

The  measures  and  submeasures  may  relate  to  various  types  of  capabilities  or  risk.  They  are 
akin  to  metrics,  but  it  is  important  to  distinguish  between  the  intended  measure’s  concept  and 
the  metric  that  is  used  to  represent  it.  For  example,  one  may  wish  to  assess  a  force  structure’s 
capability  for  “short-warning”  cases.  That  is  an  abstraction,  whereas  the  evaluation  for  a  par¬ 
ticular  planning  scenario  intended  to  represent  short-warning  cases  is  a  metric.  Such  an  evalua¬ 
tion  depends  not  only  on  the  scenario,  but  also  on  measures  of  outcome,  the  models  employed, 
and  the  detailed  inputs  to  those  models. 

Raw  Values  of  Submeasures.  We  refer  to  inputs  in  terms  of  “raw  values.”3  For  a  weapon 
system,  such  data  might  come  from  system  specifications  and  an  assumption  that  the  specifica¬ 
tions  will  be  met,  from  test  data,  from  models,  or  from  expert  judgment.  For  high-level  strate¬ 
gic  assessments  of  force  structure,  the  inputs  might  be  subjective  but  based  on  a  strong  analytic 
background  of  modeling,  war  gaming,  and  operations.  The  raw  values  may  be  expressed  in 
different  units  and  on  different  scales.  They  may  be  objective  or  subjective.  A  larger  raw  value 


3  If  Level  2  measures  are  specified  directly,  they  may  be  specified  either  as  raw  values  on  an  arbitrary  scale  or  as  scores 
between  0  and  1,  with  higher  being  better. 
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may  be  good  or  bad,  depending  on  how  the  quantity  in  question  is  defined.  Technical  risk  as 
measured  by  likelihood  of  failure  during  a  mission  is  better  if  it  is  smaller.  Technical  risk  as 
measured  by  mean  time  before  failure  is  better  if  it  is  larger. 

By  themselves,  such  raw  values  do  not  convey  a  sense  of  sufficiency.  For  that,  we  need  to 
introduce  goals  and  thresholds. 

Goals  and  Thresholds.  All  of  the  methods  used  in  PAT  except  the  Rankings  method 
involve  goals,  thresholds,  or  both.  Inputs  to  PAT  include,  for  each  submeasure,  a  raw  value 
corresponding  to  the  goal  and  a  raw  value  corresponding  to  a  threshold  (a  minimum  level  for 
perceived  utility). 

Submeasure  Scores.  PAT  calculates  the  score  of  a  submeasure  from  its  raw  value,  goal, 
and  threshold.  Except  with  the  Rankings  method  (discussed  below),  the  score  is  between  0  and 
1,  with  1  always  being  good. 

Measure  Scores.  The  score  of  a  measure  is  calculated  from  the  scores  of  its  submeasures. 
That  is,  a  measure’s  score  is  an  aggregation  of  its  submeasure  scores.4  Except  with  the  Rankings 
method,  measure-level  scores  (or,  simply,  scores)  are  defined  as  being  between  0  and  1. 

Overall-Effectiveness  Scores.  Once  PAT  has  calculated  scores  for  the  measures  charac¬ 
terizing  the  investment  options,  it  can  also  generate  the  options’  scores  for  effectiveness,  which 
is  shorthand  for  composite  or  overall  effectiveness. 

Relative  Cost-Effectiveness 

An  option’s  cost- effectiveness  is  the  ratio  of  its  effectiveness  and  cost,  but  different  costs  can  be 
used  for  the  denominator.  A  PAT  user  can  select  any  of  the  inputted  cost  categories  (e.g.,  R&D, 
acquisition,  or  O&S)  or  their  total  for  the  inputted  time  span  (e.g.,  2010-2029).  PAT  then 
calculates  an  intermediate  cost-effectiveness  for  each  option  (not  shown  in  Figure  5.1),  uses  the 
largest  value  as  the  base,  and  compares  all  options’  cost-effectiveness  to  that.  The  result,  then, 
is  the  relative  cost-effectiveness  for  each  option. 

Methods  and  Functions 

The  progression  summarized  in  Figure  5.1  requires  numerous  functions  (Fj,F2,  .  .  .  ,Fy),  as  indi¬ 
cated  in  Figure  5.1.  These  accomplish  the  following  transformations: 

•  From  sub  measure  raw  values  to  next-level  scores  (from  Level  3  to  Level  2,  and  from 
Level  2  to  Level  1). 

•  From  submeasure  scores  to  measure-level  scores  (potentially  at  both  Levels  2  and  3). 

•  From  costs  of  various  types  to  a  single  cost  used  in  cost-effectiveness  calculations. 

•  From  Level  1  scores  to  effectiveness  scores. 

•  From  effectiveness  scores  and  cost  to  cost-effectiveness. 

The  different  methods  in  PAT  to  which  we  have  alluded  use  different  functions,  as  discussed 
in  the  next  section. 


4  A  more  rigorous  term  is  “abstraction.”  A  measure  abstracts  from  or  captures  the  relevant  essence  of  the  submeasures.  It 
may  be  a  simple  average,  or  it  may  be  a  more  context-sensitive  projection.  Historically,  “aggregation”  meant  the  result  of 
collecting,  e.g.,  one  might  aggregate  the  strengths  of  nine  battalions  to  estimate  the  strength  of  a  division.  It  is  coming  to 
have  a  more  general  meaning  akin  to  “abstraction”  (Zeigler,  Praenhofer,  and  Kim,  2000). 
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Summary  of  Definitions 

Table  5.1  summarizes  the  terms  used  in  PAT  and  their  meanings  and  includes  an  example  of 
each. 


Alternative  Methods 

The  Need  for  Alternative  Methods 

Each  of  the  steps  in  Figure  5.1  requires  specifying  the  appropriate  mathematical  function.5  In 
a  given  application  of  PAT,  this  is  accomplished  when  the  analyst  chooses  a  scoring  method. 

Table  5.1 

A  Glossary  of  PAT  Terminology 


Term 


Meaning  in  PAT 


Examples 


Abstraction  A  generalization  derived  from  more 

detailed  or  concrete  cases,  perhaps  for 
a  particular  context  of  use;  measures 
are  abstractions  of  submeasures 


Aggregation 


Abstraction 


Cost-effectiveness 


The  ratio  of  an  effectiveness  score  to 
a  measure  of  cost 


Effectiveness  A  composite  score  formed  by 

abstracting  from  the  scores  of  one  or 
more  measures 


Measure 


A  way  of  evaluating  something;  a 
dimension  of  an  assessment  (similar 
to  a  metric) 


Method 


A  procedure  used  to  map  raw  scores 
into  scores  or  to  calculate  higher-level 
scores  from  lower-level  scores 


Raw  value  of  a 
submeasure 


An  unsealed  value  of  a  submeasure 


Relative  cost- 
effectiveness 


Score 


The  ratio  of  an  option's  cost- 
effectiveness  to  that  of  the  option 
with  the  highest  cost-effectiveness 

A  value  between  0  and  1  derived  from 
raw  values  and  goals  to  convey  a  sense 
of  goodness 


Engagement  effectiveness  as  derived  from  radar, 
missile,  and  kill-vehicle  effectiveness 
Effectiveness  of  an  Army  brigade  for  different 
classes  of  combat,  relative  to  a  "standard" 
brigade 

Ten-year  cost 

Overall  risk  derived  from  technical,  strategic,  and 
political  risks 


Combat  effectiveness  of  a  force  structure,  based 
on  model  outcomes  for  diverse  scenarios 


The  size  of  attack  that  saturates  a  defensive 
system 

The  likelihood  of  campaign  success  in  a  specified 
planning  scenario 

Linear  weighted  sums 
Threshold-modified  linear  weighted  sums 
Weakest  link 

The  size  of  attack  that  saturates  a  defensive 
system 

The  number  of  simultaneous  conflicts  with  which 
U.S.  forces  could  deal 


The  value  of  a  brigade  in  comparison  with  that  of 
a  "standard"  brigade 


Submeasure  One  of  the  factors  determining  a 

parent  measure 


The  size  of  attack  that  saturates  a  defensive 
system  in  a  particular  case  (e.g.,  with  attacker 
countermeasures) 


5  All  of  PAT’s  relevant  functions  are  monotonic  non-decreasing  (they  are  not  convex,  however,  as  can  be  seen  in  Figure  5.2). 
Thus,  improving  performance  of  an  investment  option  in  some  submeasure  cannot  decrease  the  effectiveness  score.  The 
aggregations  to  an  overall  effectiveness  score  have  the  same  property.  More  to  the  point,  our  scoring  methods  do  not  lead  to 
counterintuitive  conclusions  (except  in  obscure  cases  that  have  no  significance). 
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Most  work  in  mathematical  decision  analysis  uses  the  method  of  linear  weighted  sums  con¬ 
tributing  to  a  single  measure  of  “utility.”  That  method  is  taught  in  many  schoolbooks  and 
embedded  without  comment  in  much  decision-support  software.6  It  is  often  quite  useful,  but 
in  strategic  planning,  capabilities-based  planning,  and  in  much  of  systems  analysis  and  policy 
analysis  more  generally,  it  is  flawed,  for  several  reasons: 

•  Decisionmakers  need  to  know  about  some  of  the  “apples  and  oranges”  separately;  aggre¬ 
gation  into  a  single  utility  suppresses  too  much  information  and  depends  too  heavily  on 
underlying  assumptions  and  preferences  that  are  properly  in  the  decisionmaker’s  province 
rather  than  the  analyst’s.7 

•  As  decisionmakers  sometimes  observe,  “Who  knows  what  it  means  for  an  option  to 
score  0.74  rather  than  0.77?”  Such  aggregate  indexes  often  have  no  intuitive  significance 
beyond  “more  is  better.” 

•  Similarly,  decisionmakers  need  to  know  how  a  given  option  addresses  each  of  their  sepa¬ 
rate  high-level  objectives.8 

•  Aggregation  rules  sometimes  need  to  be  nonlinear  because  of  “system  effects,”  as  described 
in  the  following  examples. 


First  Example:  Ballistic  Missile  Defense.  In  reviewing  his  program,  the  Director  of  MDA  needs  to 
understand  separately  the  current  and  projected  capabilities  for  different  missions  (homeland  defense, 
defense  of  allies,  and  defense  of  U.S.  forces  deployed  abroad).  Further,  he  needs  to  understand  how 
well  the  defense  system  would  do  against  both  small  and  large  attacks  and  against  attacks  with  and 
without  various  countermeasures.  There  is  no  single  way  to  roll  all  such  information  meaningfully 
into  a  single  measure.  Further,  at  a  more  technical  level,  the  effectiveness  of  a  defense  system  for 
a  particular  mission  and  a  particular  attack  depends  in  a  nonlinear  way  on  the  effectiveness  of  the 
system's  components  (e.g.,  sensors,  interceptors,  and  kill  vehicles).  If  any  one  of  those  fails,  the  system 
fails,  regardless  of  how  well  the  other  components  perform.  Representing  such  system  issues  implies 
using  nonlinearities  in  the  mathematics  (Davis,  2002a). 

Second  Example:  Joint  and  Combined  Forces.  The  Secretary  of  Defense  might  wish  that  overall  force- 
structure  effectiveness  could  be  reduced  to  a  single  number,  but  he  needs  to  see  separately  measures 
of  the  capability  of  air,  ground,  and  sea  forces  for  a  variety  of  geographic  regions  and  operational 
circumstances.  Further,  he  knows  that  overall  warfighting  effectiveness  in  any  particular  scenario 
depends  on  the  balance  among  the  types  of  forces:  More  aircraft  cannot  compensate  for  lack  of  infantry 
in  some  irregular-warfare  scenarios,  and  more  mechanized  forces  cannot  compensate  for  ceding  air 
superiority  to  the  adversary  or  for  losing  the  ability  to  support  and  sustain  operations  through  sea- 
and  air-logistical  chains.  Again,  representing  such  system  issues  implies  using  nonlinearities  in  the 
mathematics. 


To  reflect  common  system-related  issues,  something  other  than  simple  linear  sums  is 
needed.  There  are  a  number  of  possibilities,  ranging  from  using  a  multiplicative  relationship 
instead  of  an  additive  one  to  methods  that  involve  enforcing  threshold  requirements  for  each  of 
the  critical  components.  In  PAT  terminology,  this  corresponds  to  enforcing  threshold  require¬ 
ments  for  each  submeasure  that  characterizes  an  investment  option. 


6  The  classic  introductory  book  on  decision  analysis  (Raiffa,  1968)  is  quite  readable.  A  later  text  treats  multiobjective  deci¬ 
sion  analysis  (Keeney  and  Raiffa,  1976).  Although  it  emphasizes  combining  sums  into  a  single  utility,  the  book  discusses 
alternatives  to  simple  linear  weighted  sums. 

7  This  consideration  has  led  to  policy  analysis’  emphasis  on  scorecards.  An  early  RAND  application — to  a  Netherlands 
water-management  problem  (Goeller  et  ah,  1983) — was  particularly  influential  in  causing  scorecards  to  be  adopted. 

8  Value-focused  thinking  is  a  form  of  multiobjective  decision  analysis  that  organizes  around  an  organization’s  objectives 
(Keeney,  1992).  It  has  been  used  in  a  number  of  military  applications  (see,  e.g.,  Parnell,  2006). 
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PAT  has  five  built-in  methods  for  aggregation  of  scores.  This  greatly  increases  flexibility, 
bur  it  also  increases  complexity  and  can  undercut  the  goal  of  having  relatively  simple,  logical, 
and  intuitive  results.  Thus,  the  analyst  should  decide  which  method  or  methods  to  use  and 
then  present  only  those.  The  five  methods  (and  their  short  names)  are  as  follows:  (1)  goal-based 
(Goals);  (2)  goal-based  with  weak  thresholds  (Weak  Thresholds),  (3)  goal-based  with  thresh¬ 
olds  (Thresholds),  (4)  goal-based  with  weakest  link  (Weakest  Link),  and  (5)  rankings-based 
(Rankings). 

These  are  described  one  by  one  below  and  are  then  summarized  in  Table  5.5  on  p.  65.  For 
each  method,  the  description  specifies  how  scores  are  generated  from  the  raw  values  of  submea¬ 
sures  and  how  higher-level  scores  and  effectiveness  are  calculated  by  aggregation. 

We  use  a  common  notation,  as  defined  in  Table  5.2.  The  notation  and  subsequent  discus¬ 
sion  apply  only  to  problems  limited  to  Level  1  and  Level  2  measures  but  can  easily  be  general¬ 
ized  to  the  case  that  has  Level  3  measures  as  well. 

Goals  Method 

The  Goals  method  is  the  simplest  to  describe.  Every  measure  is  composed  of  a  collection  of 
submeasures,  each  of  which  has  a  goal  value  that  each  investment  option  is  trying  to  achieve. 

Submeasure  Scores  with  Goals  Method.  A  submeasure’s  score  is  0  or  G. ,,  depending 
on  whether  or  not  the  raw  value  has  reached  the  goal.  That  is,  for  any  investment  option  i  and 
any  measure  j,  if  the  scale  is  increasing  so  that  goals  correspond  to  high  values,  then  the  /?th 
submeasure’s  score  is  given  by 


^  =  0ifk^<^ 

^ijjc  ~ G jkifVi  jk  > Vjj' 

If  goodness  increases  with  decreasing  raw  values,  the  equations  change  accordingly. 

Aggregation  to  Find  Measure  Scores  with  Goals  Method.  The  function  used  to  calcu¬ 
late  the  score  of  a  measure  is  just  a  weighted  sum  of  the  submeasure  scores: 

k=\ 


If  all  submeasures  are  equally  weighted,  the  measure  score  is  the  fraction  of  the  mea¬ 
sure’s  submeasures  that  reached  their  goals.  The  resulting  measure  scores  are  between  0  and 
1  because  of  the  normalization  accomplished  by  the  denominator.  In  practice,  with  PAT,  the 
values  of  G.k  will  usually  be  set  to  1,  with  the  W  k  values  establishing  the  relative  weights  of  the 
submeasures,  but  for  the  sake  of  completeness,  we  include  G  k  in  our  equations. 
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Table  5.2 

Notation  for  Defining  Scoring  Methods 


Symbol 

Meaning 

Source 

Uj,k 

Indexes  for  investment  option,  measure,  and 
submeasure,  respectively 

n/a 

m 

Number  of  measures 

n/a 

ni 

Number  of  submeasures  of  measure  j 

n/a 

Raw  values  of  the  submeasures  for  investment 
option  i  and  measure  j 

Level  2  or  Level  3  Data 

{V'V-Vl 

Scores  of  the  n  submeasures  (submeasure  scores) 
for  investment  option  /  and  measure  j 

Calculated 

Scores  of  the  m  measures  (measure  scores)  for 
investment  option  / 

Calculated 

Et 

Overall  effectiveness  of  investment  option  / 

Calculated 

Threshold  raw  values  for  each  submeasure  of 
measure  j 

Level  2  or  Level  3  Data 

Corresponding  goal  raw  values  for  each  submeasure 
of  measure  j 

Level  2  or  Level  3  Data 

f  rpG  n-,G  rj-,G  1 

V  y.1* 1  jl""1  j,n  J 

Submeasure  scores  when  submeasure  raw  values 
have  reached  thresholds  under  measure  j 

Level  2  or  Level  3  Data 

{g,,,gix,...g,„} 

Scores  of  submeasures  that  have  reached  their  goals 
under  measure  j 

Level  2  or  Level  3  Data 

{w^w^.wl} 

Weights  of  the  submeasures  in  computing  measure 
scores 

Level  2  or  Level  3  Data 

{cl,c1,...cm} 

Weights  of  the  measures  in  computing  effectiveness 

Level  1  Data 

Aggregation  to  Find  Overall  Effectiveness  with  Goals  Method.  The  function  used  to 
set  the  overall  effectiveness  under  investment  option  i  is  also  a  weighted  sum  over  measures: 


m 


7=1 


The  measure  and  effectiveness  scores  all  have  values  between  0  and  1.  The  weights  need  not  be 
between  0  and  1,  since  PAT  performs  normalizations,  but  it  is  arguably  good  practice  to  enter 
weights  in  that  range  that  add  up  to  1  so  that  the  significance  of  a  given  weight’s  value  will  be 
more  readily  understood  without  the  user  having  to  look  at  the  other  weights  and  do  a  mental 
calculation. 
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Weak  Thresholds  Method 

Submeasure  Scores  with  Weak  Thresholds  Method.  With  the  Weak  Thresholds  method, 
each  submeasure  has  a  threshold  and  a  goal.  If  the  threshold  is  not  reached,  the  submeasure 
score  is  0.  If  the  submeasure  reaches  or  exceeds  its  goal,  the  submeasure  score  is  its  score  for  the 
goal  value.  In  between,  the  score  is  determined  by  a  linear  relationship.  That  is,  in  cases  where 
goodness  increases  with  raw  value: 


sIJJk= o 

Sjjjt  =  Gjk 


5.. 


V  u-T 

-  y  |  -1 

j,k  yG  — yT 

V  j,k  V  jjk 


^(g.,-T.,\ if  VT,  <V  ,<V 

V  J*  ]’k)  J’k  '-J’k 


G 

}* 


If  goodness  decreases  with  raw  value,  the  equations  must  be  adjusted.  Figure  5.2  shows 
graphically  the  resulting  scoring  method  for  cases  in  which  more  is  better  (solid  line)  and  cases 
for  which  more  is  worse  (dashed  lines).  The  order  of  goal  and  threshold  is  reversed  for  the  two 
cases — for  the  more-is-worse  case  (dashed  lines),  the  threshold  comes  second  and  the  goal  first 
as  one  reads  along  the  x-axis  (see  the  parenthetical  values). 

Figure  5.2 

Score  Versus  Raw  Value  for  Goals  and  Thresholds  Methods 


Score  Score 
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Aggregation  to  Find  Measure  Scores  with  Weak  Thresholds  Method.  The  score  of 
measure  7  is  again  given  by  a  linear  weighted  sum,  as  in  the  Goals  method,  but  the  submeasure 
scores  are  different: 


2Xa, 


k= i 


t,J  8, 


k=l 


Aggregation  to  Find  Overall  Effectiveness  with  Weak  Thresholds  Method.  Overall 
effectiveness  with  the  Weak  Thresholds  method  is  given  by  a  linear  weighted  sum,  as  previ¬ 
ously,  but  the  measure  scores  are  different: 


It  Pi., 


m 

If, 


M 


Thresholds  Method 

Submeasure  Scores  with  Thresholds  Method.  Submeasure  scores  are  calculated  with 
the  same  function  as  that  used  for  the  Weak  Thresholds  method: 


5. 


S,,j,k-Gj,k 


s 


■-T..+ 

J,k 


V  -T 

V  i,j,k  ±  j,k 

iTT 


G 

j,k 


Aggregation  to  Find  Measure  Scores  with  Thresholds  Method.  The  difference  in 
methods  occurs  here.  If  any  submeasure  fails  to  reach  its  threshold,  the  measure  is  assigned  a 
score  of  0: 


M.  ,  =  — - if  all  submeasures  reach  their  thresholds 

i,j  tij 

I 

k=\ 

M.  =0  otherwise 

l'J 


Aggregation  to  Find  Overall  Effectiveness  with  Thresholds  Method.  Effectiveness  is 
calculated  precisely  as  before,  but  the  measure  scores  are  different: 
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7=1 


Weakest  Link  Method 

The  Weakest  Link  Method  is  even  more  stringent  in  enforcing  the  “requirements”  represented 
by  thresholds. 

Submeasure  Scores  with  Weakest  Link  Method.  Submeasure  scores  are  calculated 
with  the  same  function  as  for  the  other  threshold  methods: 


■*,,,*  =0 
Si.jJ r  =  GjJk 


«vjM>vlk 


s, 


■  -  Ti* + 


V  -T 

r  i,j,k  x  j,k 
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Aggregation  to  Find  Measure  Scores  with  Weakest  Link  Method.  With  the  Weakest 
Link  Method,  the  measure  score  is  the  minimum  of  the  submeasure  scores.  If  any  submeasure 
fails  to  reach  its  threshold,  the  measure  score  will  be  0,  as  in  the  Thresholds  method,  but  if 
all  submeasures  reach  their  thresholds,  the  score  will  be  different  from  that  in  the  Thresholds 
method  and  typically  smaller: 


Aggregation  to  Find  Overall  Effectiveness  with  the  Weakest  Link  Method.  Aggre¬ 
gation  with  the  Weakest  Link  method  is  performed  simply  by  taking  the  minimum  measure 
score  (which  is  identical  to  the  minimum  submeasure  score  of  all  submeasures).  If  any  submea¬ 
sure  in  any  of  the  measures  fails  to  reach  its  threshold,  effectiveness  will  be  0.  Even  if  that  does 
not  happen,  effectiveness  will  typically  be  smaller  in  this  method: 


Rankings  Method 

The  Rankings  method  does  not  use  goal  or  threshold  values.  Instead,  for  each  submeasure,  the 
investment  options  are  simply  ranked  from  best  to  worst,  without  regard  to  absolute  perfor¬ 
mance.  Instead  of  submeasure  and  measure  scores,  we  refer  to  submeasure  and  measure  ranks 
for  this  method. 

Submeasure  Ranks.  Let  Rijk  be  the  rank  of  investment  option  i  for  measure  j  and  for 
submeasure  k.  We  define  it  as  one  more  than  the  number  of  investment  options  that  perform 
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strictly  better  than  investment  option  i  on  submeasure  £of  measure/  Thus,  if  two  investment 
options  have  the  same  raw  value,  they  will  have  the  same  rank. 

Aggregation  to  Find  Measure  Ranks  with  Rankings  Method.  The  aggregation  from 
submeasure  rankings  to  measure  rankings  is  again  a  linear  weighted  sum.  We  calculate  the 
average  rank  Rjj  within  measure  7  as 


X,r  *• 


&.;  = 


k=l 


2X 


t=i 


The  value  Rij  is  used  to  set  the  color  (one  of  five  shades  of  blue,  as  shown  in  Table  5.4  on 
p.  64)  in  the  measure-summary  table,  depending  on  the  quintile  in  which  the  average  ranking 
resides. 

Aggregation  to  Find  Overall  Effectiveness  with  Rankings  Method.  This  aggregation 
function  is  best  explained  with  words  and  examples.  It  is  entirely  different  from  the  functions 
used  in  the  other  methods.  For  this  effectiveness  aggregation,  each  measure  is  assigned  a  score 
based  on  rankings. 

Let  us  assume  that  K  investment  options  are  under  consideration  and  that  each  submea¬ 
sure  has  K(K—  \)/2  points  to  distribute  among  the  investment  options.  If  there  are  no  ties  for 
the  values  in  submeasure/  the  submeasure  score  is  given  by 


R'..=K- 

l,j,k 


R 


,j,k 


If  there  are  ties,  the  points  that  would  have  gone  to  the  investment  options  in  those  posi¬ 
tions  are  combined  and  then  equally  distributed  among  the  investment  options.  For  example, 
with  ten  investment  options,  the  second-  and  third-place  investment  options  would  receive 
8  and  7  points,  respectively.  If  two  investment  options  are  tied  for  second  place,  they  would 
receive  (7+8)/2  =  7.5  points  each.  This  scoring  method  is  used  instead  of  a  linear  transformation 
of  the  rankings  to  avoid  producing  artificially  large  numbers  in  the  case  of  ties.  To  illustrate, 
if  each  of  ten  investment  options  had  the  same  value  for  a  submeasure,  they  would  all  tie  for 
first.  If  the  ranking  effectiveness  score  did  not  take  ties  into  account,  each  investment  option 
would  receive  9  points,  meaning  that  the  total  (unweighted)  contribution  to  the  final  score 
(summed  across  investment  options)  by  that  submeasure  would  be  90  points.  By  comparison, 
for  a  submeasure  where  every  investment  option  had  a  different  value,  the  total  (unweighted) 
contribution  to  the  final  score  by  that  submeasure  would  be  45  points  (i.e.,  9  +  8  +  7  +  6  +  5 
+  4  +  3  +  2  +  1). 

As  with  the  other  scoring  methods,  each  submeasure  S ,  has  a  weight  W  k  in  the  aggrega¬ 
tion  to  the  measure  score.  Similarly,  the  weights  C  determine  the  relative  contribution  of  each 
measure  to  the  effectiveness  score  for  each  investment  option.  The  measure  score  for  an  invest¬ 
ment  option  i  for  measure  j  is 


Details  of  the  Methodology  63 


M.,= 


IXx,< 

k=\ 


Because  the  maximum  possible  value  of  Rij  for  each  individual  submeasure  is  K -  1,  the 
denominator  in  the  above  expression  scales  the  effectiveness  score  to  be  between  0  and  1 .  The 
effectiveness  score  for  each  investment  option  over  all  measures,  denoted  E.,  is 


ICjK.j 


M 


Color-Coding  in  Scorecards 

On  scorecards,  PAT  represents  the  scores  of  measures  or  submeasures  by  colors  (or  by  a  com¬ 
bination  of  colors  and  numerical  values,  if  desired).  The  conventions  used  for  the  color-coding 
are  of  two  types,  one  for  the  Thresholds,  Weak  Thresholds,  and  Weakest  Link  methods  and 
one  for  the  Rankings  method. 

Colors  for  Thresholds,  Weak  Thresholds,  and  Weakest  Link  Methods 

For  all  the  non-Rankings  methods  used  by  PAT,  the  scores  of  the  investment  options  can  be 
mapped  into  the  colors  of  familiar  stoplight  charts,  where  red  is  worst  and  orange,  yellow,  light 
green,  and  green  are  successively  better.  A  five-color  system  is  used  for  our  measure-summary 
table,  because  over  the  decades,  five  has  proven  to  be  a  comfortable  number  that  makes  suffi¬ 
cient  distinctions  but  avoids  cognitive  overload.  The  mapping  for  the  measure  scores  is  shown 
in  Table  5.3.  In  addition,  for  the  Thresholds  and  Weakest  Link  methods,  if  any  submeasure 
fails  to  reach  a  threshold,  the  cell  in  the  measure  summary  may  optionally  have  an  F  shown  in 
the  upper  right-hand  corner.  The  convention  in  PAT  is  that  the  score  leans  upward  at  boundar¬ 
ies,  so  that  a  score  of  0.800000  is  dark  green,  whereas  a  score  of  0.799999  is  light  green. 

Colors  for  Rankings  Method 

Color-coding  on  the  measure-summary  table  for  the  Rankings  method  is  different  and  even 
uses  different  colors,  to  avoid  conveying  the  impression  of  good  and  bad  associated  with  the 
stoplight  charts  for  the  Goals  and  Thresholds  methods. 
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Table  5.3 

Mapping  Measure  Scores  into  Colors 


Range  of  Scores 

Color 

0  <  score  <  0.2 

0.2  <  score  <  0.4 

Orange 

0.4  <  score  <  0.6 

Yellow 

0.6  <  score  <  0.8 

Light  green 

0.8  <  score  <  1 

Dark  green 

As  shown  in  Table  5.4,  the  colors  go  from  light  blue  to  dark  blue,  denoting  the  quintile  in 
which  the  weighted  average  rank  of  the  investment  option  lies.  For  example,  if  there  were  ten 
investment  options,  an  investment  option  would  receive  the  lightest  blue  color  if  the  weighted 
average  rank  was  two  or  lower  and  would  receive  the  darkest  blue  color  if  it  was  eight  or  greater. 
The  same  coloring  method,  based  on  quintiles,  is  used  on  the  Drilldown  sheets,  where  the 
rankings  in  each  submeasure  set  the  color  for  the  cell. 

Table  5.5  summarizes  the  various  methods  concisely. 


Examples  of  Scoring  and  Aggregation  Using  Different  Methods 

To  provide  examples  for  each  scoring  method,  we  look  at  two  measures,  M  and  M},  each 
consisting  of  three  submeasures.  All  sub  measures  take  values  between  0  and  10,  inclusive, 
with  2  being  the  threshold  value  and  7  being  the  goal  value  in  each  case.  For  both  measures, 
submeasure  S.  will  have  weights  twice  those  of  the  other  two  submeasures;  all  submeasures 
receive  a  score  of  0  for  not  reaching  the  threshold  value.  By  setting  the  weights  of  the  three 
submeasures  to  0.25,  0.25,  and  0.5,  we  get  a  sum  of  1,  so  there  is  no  need  to  divide  the  sum  of 
the  submeasure  scores  by  the  sum  of  the  weights.  We  also  assume  the  weights  of  the  measures 
are  equal.  We  consider  three  investment  options,  which  we  call  Investment  Options  A,  B,  and 
C.  The  raw  values  for  each  investment  option  for  each  submeasure  are  given  in  Table  5.6.  (This 
and  subsequent  tables  in  this  chapter  are  not  PAT  displays,  but  rather  were  constructed  for  the 
discussion.) 


Table  5.4 

Color-Coding  in  the  Rankings  Method 


Meaning 

Color 

(Average)  rank  in  first  quintile 

(Average)  rank  in  second  quintile 

(Average)  rank  in  third  quintile 

(Average)  rank  in  fourth  quintile 

(Average)  rank  in  fifth  quintile 
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Table  5.5 

Summary  of  Methods 


Method 

Submeasure  Scores 

Measure  Scores 

Overall 

Effectiveness 

Coloring 

Method 

Comment 

Goals 

0  or  1,  depending 
on  whether  goal  is 
reached 

Linear  weighted  sum  of 
submeasure  scores 

Linear  weighted 
sum  of  measure 

scores 

As  in  Table  5.3 

Simple  and  common 
but  arguably 
simple-minded 

Weak 

Thresholds 

As  in  Figure  5.2 

Linear  weighted  sum  of 
submeasure  scores 

Linear  weighted 
sum  of  measure 

scores 

As  in  Table  5.3 

May  be  appropriate 
if  not  all 

submeasures  are 
critical 

Thresholds 

As  in  Figure  5.2 

Zero  if  any  submeasure 
fails  to  reach  threshold; 
otherwise,  linear 
weighted  sum  of 
submeasure  scores 

Linear  weighted 
sum  of  measure 

scores 

As  in  Table  5.3 

May  be  appropriate 
if  all  submeasures 
are  critical  and  have 
firm  requirements 

Weakest 

Link 

As  in  Figure  5.2 

Minimum  of  submeasure  Minimum  of 
scores  measure  scores 

As  in  Table  5.3 

May  be  appropriate 
if  all  measures  and 
submeasures  are 
critical  and  have 
firm  requirements 

Rankings 

Modified  Borda 
count 

Weighted  average  of 
ranks  for  measure- 
summary  table,  linear 
weighted  sum  of  Borda 
scores  for  effectiveness 

Linear  weighted 
sum  of  measure 

scores 

As  in  Table  5.4 

May  be  appropriate 
if  one  wishes  to 
avoid  discussion 
of  goals  and 
thresholds 

Table  5.6 

Illustration  of  Scoring  Methods 


Raw  Value 


Submeasure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

M1.1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values  desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  value 

7 

7 

7 

7 

7 

7 

Level  2  value  for  Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Goals  Method 

Table  5.7  presents  illustrative  results  for  the  Goals  method.  Values  that  meet  or  exceed  the 
goal  value  are  highlighted  in  green.  The  table  also  shows  the  calculation  of  the  submeasure  and 
measure  scores,  and  the  measure-score  cell  for  each  measure  is  colored  to  correspond  to  the 
color  scheme  that  appears  on  the  Summary  sheet  in  PAT. 
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Table  5.7 

Illustrative  Results  for  the  Goals  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

M1.1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values 
desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  value 

7 

7 

7 

7 

7 

7 

Level  2  value  for 
Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Ml 

score 

M2 

score 

Effec¬ 

tiveness 

Option  A 

10 

10 

5 

5 

5 

5 

Weighted  score 

.25 

.25 

0 

.5 

0 

0 

0 

.25 

Option  B 

7 

5 

6 

8 

1 

10 

Weighted  score 

.25 

0 

0 

.25 

.25 

0 

.5 

.75 

.5 

Option  C 

1 

5 

10 

10 

10 

10 

Weighted  score 

0 

0 

.5 

.5 

.25 

.25 

.5 

1 

.75 

Since  the  weights  of  the  measures  are  equal,  the  effectiveness  score  is  just  the  average  of 
the  two  measure  scores.  Although  it  reaches  only  one  goal  in  measure  M ,  that  submeasure  is 
weighted  twice  as  much  as  the  other  two,  hence  it  has  the  same  weighted  percentage  of  goals 
reached  as  Investment  Option  A,  which  reaches  two  goals  in  measure  M f. 

Thresholds  Method 

With  the  Thresholds  method,  values  for  measures  that  meet  or  exceed  the  goal  value  are  high¬ 
lighted  in  green;  values  that  meet  or  exceed  the  threshold  value  (but  not  the  goal  value)  are 
highlighted  in  yellow;  and  values  that  fail  to  meet  the  threshold  value  are  highlighted  in  red 
(Table  5.8).  In  addition,  we  show  the  calculation  of  the  measure  score  and  effectiveness  score 
and  color  the  summary-score  cell  for  each  measure  to  correspond  to  the  color  scheme  that 
appears  on  the  Summary  sheet  in  PAT  for  the  Thresholds  method.  We  also  show  the  effec¬ 
tiveness  score  (assuming  the  measures  have  equal  weight)  for  both  measures  in  the  rightmost 
column.  In  a  weighted-score  row,  a  cell  containing  xxx  means  that  the  submeasure  failed  to 
reach  the  threshold  value,  so  the  measure  score  for  the  entire  measure  is  zero. 

If  an  investment  option  reaches  the  goal,  the  weighted  submeasure  score  is  1  times  the 
submeasure  weight.  If  the  raw  value  is  between  the  threshold  and  the  goal  (say,  5),  because  5 
is  60  percent  of  the  way  from  the  threshold  (2)  to  the  goal  (7),  the  unweighted  submeasure 
score  would  be  0.6,  which  is  then  multiplied  by  the  submeasure  weight  to  get  the  weighted 
submeasure  score.  In  this  case,  Investment  Option  A  did  not  fail  to  reach  any  of  the  threshold 
values,  so  it  has  the  highest  effectiveness  score,  even  though  Investment  Option  C  meets  four 
out  of  six  goals. 
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Table  5.8 

Illustrative  Results  for  the  Thresholds  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

Ml  .1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values 
desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  value 

7 

7 

7 

7 

7 

7 

Level  2  value  for 
Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Ml 

score 

M2 

score 

Effec¬ 

tiveness 

Option  A 

10 

10 

5 

5 

5 

5 

Weighted  score 

.25 

.25 

.3 

.8 

.15 

.15 

.3 

.6 

.7 

Option  B 

7 

5 

6 

8 

10 

Weighted  score 

.25 

.15 

.4 

.8 

.25 

XXX 

Ln 

1 

.4 

Option  C 

5 

10 

10 

10 

10 

Weighted  score 

XXX 

.15 

.5  1 

.25 

.25 

.5 

1 

.5 

Weak  Thresholds  Method 

For  the  Weak  Thresholds  method,  values  that  meet  or  exceed  the  goal  value  are  highlighted 
in  green;  those  that  meet  or  exceed  the  threshold  value  but  not  the  goal  value  are  highlighted 
in  yellow;  and  those  that  fail  to  meet  the  threshold  value  are  highlighted  in  red  (Table  5.9). 
We  also  show  the  measure  score  and  effectiveness  score  and  color  the  cells  appropriately.  We 
show  the  effectiveness  score  (assuming  the  measures  have  equal  weight)  for  both  measures  in 
the  rightmost  column.  Even  though  Investment  Option  C  fails  for  submeasure  S  |5  reaching 
the  goal  value  on  four  of  six  submeasures  pushes  its  effectiveness  above  that  of  the  other  two 
options. 

Weakest  Link  Method 

For  the  Weakest  Link,  values  that  meet  or  exceed  the  goal  value  are  highlighted  in  green;  those 
that  meet  or  exceed  the  threshold  value  but  not  the  goal  value  are  highlighted  in  yellow;  and 
those  that  fail  to  meet  the  threshold  value  are  highlighted  in  red  (Table  5.10).  We  also  show  the 
(unweighted)  submeasure  score,  the  measure  score  (which  is  the  minimum  of  the  submeasure 
scores  for  each  measure),  and  the  effectiveness  score  (which  is  the  minimum  of  the  measure 
scores)  for  each  investment  option.  The  weights  of  the  submeasures  do  not  apply  here.  We  color 
the  effectiveness  score  cell  for  each  measure  to  correspond  to  the  color  scheme  used  in  PAT  for 
this  method.  Because  Investment  Option  A  is  the  only  option  that  did  not  fail  on  any  submea¬ 
sure,  it  has  the  highest  effectiveness  score  under  the  Weakest  Link  method. 
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Table  5.9 

Illustrative  Results  for  the  Weak  Thresholds  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

M1.1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values 
desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  value 

7 

7 

7 

7 

7 

7 

Level  2  value  for 
Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Ml 

score 

M2 

score 

Effec¬ 

tiveness 

Option  A 

10 

10 

5 

5 

5 

5 

Weighted  score 

.25 

.25 

.3 

.8 

.15 

.15 

.3 

.6 

.7 

Option  B 

7 

5 

6 

8 

10 

Weighted  score 

.25 

.15 

.4 

.8 

.25 

0 

.5 

.75 

.4 

Option  C 

5 

10 

10 

10 

10 

Weighted  score 

0 

.15 

.5 

.65 

.25 

.25 

.5 

1 

.83 

Table  5.10 

Illustrative  Results  for  the  Weakest  Link  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

M1.1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values 
desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  Value 

7 

7 

7 

7 

7 

7 

Level  2  value  for 
Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Ml 

score 

M2 

score 

Effec¬ 

tiveness 

Option  A 

10 

10 

5 

5 

5 

5 

Weighted  score 

1 

1 

.6 

.6 

.4 

.4 

.4 

.4 

.4 

Option  B 

7 

5 

6 

10 

Weighted  score 

1 

.4 

.6 

.4 

1 

0 

1 

5 

10 

10 

10 

10 

Weighted  score 

0 

.4 

1 

1 

1 

1 

0 
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Rankings  Method 

In  our  example,  the  summary  and  effectiveness  weights  are  identical  (with  the  third  submea¬ 
sures  of  each  measure  having  twice  the  weight  of  the  other  two),  as  shown  in  Table  5.11. 

The  average  rank-  and  measure-score  calculations  are  compressed  slightly,  and  all  values 
are  rounded  to  two  decimal  places.  For  the  measure  score,  the  sum  of  the  submeasure  scores  is 
divided  by  the  product  of  the  maximum  submeasure  score  per  measure  (2)  and  the  sum  of  the 
weights  (1).  The  effectiveness  value  is  just  the  average  of  the  measure  scores.  Because  Invest¬ 
ment  Option  C  has  the  best  value  for  four  of  six  submeasures,  it  is  clearly  the  best  option  under 
the  Rankings  method. 


Table  5.11 

Illustrative  Results  for  the  Rankings  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Level  2  measure 

M1.1 

Ml. 2 

Ml. 3 

M2.1 

M2. 2 

M2. 3 

Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

High  or  Low  values 
desired? 

High 

High 

High 

High 

High 

High 

Threshold  value 

2 

2 

2 

2 

2 

2 

Goal  value 

7 

7 

7 

7 

7 

7 

Level  2  value  for 
Threshold 

0 

0 

0 

0 

0 

0 

Level  2  value  for  Goal 

1 

1 

1 

1 

1 

1 

Avg. 

rank 

Avg. 

rank 

Effec¬ 

tiveness 

Option  A 

10 

10 

5 

5 

5 

5 

(rank)x(weight) 

.25 

.25 

1.5 

2 

.75 

.5 

1.5 

2.75 

(score)x(weight) 

.5 

.5 

0 

.5 

0 

.25 

0 

.13 

.32 

Option  B 

7 

5 

6 

8 

1 

10 

(rank)x(weight) 

.5 

.5 

1 

2 

.5 

.75 

.5 

1.75 

(score)x(weight) 

1 

.125 

.5 

.44 

.25 

0 

1 

.63 

.54 

Option  C 

1 

5 

10 

10 

10 

10 

(rank)x(weight) 

.75 

.5 

.5 

1.75 

.25 

.25 

.5 

1 

(score)x(weight) 

.25 

.125 

1 

.69 

.5 

.5 

1 

1 

.85 

CHAPTER  SIX 


Marginal  and  Chunky  Marginal  Analysis 


Introduction 

In  marginal  analysis,  small  changes  to  key  variables  in  a  system  are  considered  one  at  a  time. 
Marginal  analysis  is  often  used  to  find  which  variables  are  most  responsible  for  affecting  the 
outcomes  of  a  system.  When  the  key  variables  are  investments,  marginal  analysis  helps  to 
determine  what  “the  next  dollar”  (or  million  dollars)  should  be  allocated  to  (or  removed  from) 
in  order  to  maximize  the  capability  of  the  resulting  collection  of  investments.  Ideally,  the  small 
changes  in  each  variable  are  equivalent  in  magnitude,  to  enable  relevant  comparisons  between 
options.  Although  marginal  analysis  can  be  used  for  a  variety  of  systems,  we  restrict  our  discus¬ 
sion  to  the  marginal  analysis  of  investments. 

Marginal  analysis  is  not  appropriate  for  all  situations.  For  example,  small  deviations  in 
investments  may  have  no  effect  on  system  outputs.  This  happens  particularly  when  invest¬ 
ments  are  tied  to  purchases  of  discrete  objects.  An  extra  dollar  invested  in  the  acquisition  of  a 
radar  system  has  absolutely  no  impact  when  a  single  component  costs  thousands  or  millions  of 
dollars.  Similarly,  investments  in  systems  that  require  a  large  buy-in  before  they  become  effec¬ 
tive  have  no  impact  until  the  buy-in  is  reached,  at  which  point  there  may  be  a  large  discontinu¬ 
ity  in  capability. 

Conversely,  small  deviations  in  investments  may  have  a  disproportionate  effect  on  the 
outputs  of  the  system.  A  budget  that  includes  90  percent  of  the  cost  to  acquire  a  missile  does 
not  get  the  investor  90  percent  of  a  missile.  For  the  most  part,  small  reductions  in  investment 
do  not  lead  to  proportionate  reductions  in  system  capability. 

Marginal  analysis  tends  to  be  more  meaningful  when  the  options  that  result  from  con¬ 
stant  deductions  to  each  investment  are  really  of  equal  value.  Consider  a  situation  in  which 
$16  million  is  spent  to  acquire  three  different  types  of  missiles,  where  each  individual  missile 
costs  $1  million.  Assume  further  that  the  current  investment  plan  purchases  one  missile  of 
the  first  type,  five  of  the  second  type,  and  ten  of  the  third  type.  Three  equal-cost  investment 
options  that  could  result  from  a  $1  million  cut  in  funding  correspond  to  not  purchasing  one 
missile  of  each  type.  In  contrast,  consider  a  situation  where  $16  million  is  spent  in  the  acquisi¬ 
tion  of  three  missiles,  one  costing  $1  million,  one  costing  $5  million,  and  one  costing  $10  mil¬ 
lion.  A  cut  of  $1  million  from  any  of  the  three  missile  purchases  results  in  not  getting  any  mis¬ 
siles,  so  this  cut  actually  results  in  comparing  three  cases  costing  $6  million,  $11  million,  and 
$15  million  for  the  acquisition  of  two  of  the  three  types  of  missiles. 

This  suggests  a  variant  of  marginal  analysis  in  which  the  changes  to  the  current  set  of 
investments  represent  the  removal  (or  addition)  of  whole  purchases.  As  with  marginal  analy¬ 
sis,  these  purchases  are  removed  individually  from  the  current  investment,  with  each  possible 
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removal  defining  a  new  investment  option.  The  costs  of  the  resulting  investment  cases  provide 
another  measure  that  can  be  used  as  a  basis  for  comparison.  The  important  question  is  often 
not,  “Where  do  we  spend  the  next  dollar?”  but  rather,  “How  do  we  invest  this  extra  $50  mil¬ 
lion?”  (Pessimists  or  realists  will  suggest  that  the  question  asked  more  often  is,  “How  do  we 
handle  this  $50  million  budget  cut?”)  This  chunky  marginal  analysis  method  will  be  described 
in  more  detail  below,  as  will  the  application  of  PAT  to  assist  with  this  analysis. 


Chunky  Marginal  Analysis  for  a  Ballistic  Missile  Defense  Example 

We  consider  as  a  base  case  a  fixed  collection  of  ballistic-missile  defense  system  (BMDS)  invest¬ 
ments  over  time  in  both  R&D  and  acquisitions.  We  generate  a  collection  of  investment  options 
by  considering  variations  from  the  base  case.  These  steps  are  large-scale  additions  to  (or  sub¬ 
tractions  from)  the  base  case.  Some  examples  would  be  the  cancellation  of  R&D  on  a  particu¬ 
lar  program,  or  a  reduced  (or  increased)  acquisition  of  a  radar  system  or  interceptor.  The  steps 
should  not  be  so  small  that  it  is  impossible  to  discern  the  base  case  from  the  BMDS  associated 
with  the  new  investment  option,  but  they  should  not  be  so  large  that  one  could  be  decomposed 
into  a  collection  of  smaller,  meaningful  steps.  Thus,  a  step  should  not  consist  of  the  cancella¬ 
tion  of  three  unrelated  programs;  rather,  three  new  options  should  be  defined  by  the  cancel¬ 
lation  of  each  individual  program.  In  addition,  a  step  should  be  maximal  in  the  sense  that 
any  program  that  is  made  unnecessary  (or  necessary)  by  the  cancellation  (or  addition)  of  one 
program  should  also  be  cut  (or  included)  along  with  that  program.  For  example,  cutting  the 
development  of  a  radar  platform  should  also  cut  the  development  of  any  battle-management 
suite  associated  with  that  radar  platform  (unless  the  suite  can  also  be  used  for  other  radar  sys¬ 
tems  under  development). 

Once  these  steps  are  defined,  each  investment  option  consists  of  the  base  case  and  one 
(or  more)  of  these  steps.  As  the  number  of  steps  increases,  the  number  of  possible  investment 
options  increases  exponentially.  It  may  be  best  to  restrict  analysis  to  options  that  are  at  most 
a  fixed  number  of  steps  away  from  the  base  case.  With  n  possible  steps,  there  are  roughly  n2/ 2 
options  that  are  at  most  two  steps  away  from  the  base  case  and  about  n3/S  options  that  are  at 
most  three  steps  away  from  the  base  case.  The  number  of  investment  options  under  consider¬ 
ation  should  also  be  tempered  by  the  ability  to  determine  the  costs  associated  with  each,  as  well 
as  the  ability  to  analyze  the  BMDS  that  results  from  each.  PAT  can  store  thousands  of  invest¬ 
ment  cases  and  can  display  some  or  all  of  those  cases  on  its  output  sheets. 

The  output  sheet  that  is  perhaps  most  useful  in  assisting  with  chunky  marginal  analysis  is 
the  Scatter  Plot,  where  the  user  can  select  the  x-axis  from  a  collection  of  cost  metrics  (over  vari¬ 
ous  time  periods,  constrained  to  R&D  investment  or  deployment  investment  only,  etc.)  and 
the  y-axis  from  two  different  evaluation  metrics.  Investment  options  are  plotted  on  the  display 
as  points  of  different  colors  (and  different  shapes  if  more  than  one  measure  is  displayed  on  the 
y-axis).  This  is  particularly  useful  in  determining  which  investment  option  is  best  (and  the  cor¬ 
responding  programs  that  should  be  cut)  if  the  budget  is  reduced  from  the  base  case. 

Consider  a  notional  example  of  a  base  case  that  consists  of  investments  in  three  options 
(called  “engagement  sequence  groups”  at  MDA):  one  boost  phase,  one  midcourse  phase,  and 
one  terminal  phase  (based  in  the  United  States).  There  are  three  obvious  steps  away  from  this 
base  case:  the  cancellation  of  programs  specific  to  each  phase  of  the  defense  (in  general,  defense 
systems  from  different  phases  may  share  tracking  systems,  so  a  cut  of  a  particular  phase  of  the 
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defense  does  not  necessarily  eliminate  all  programs  associated  with  that  phase).  Because  the 
number  of  steps  is  so  small,  there  is  no  reason  to  not  consider  the  eight  investment  options  cor¬ 
responding  to  all  the  subsets  of  the  phases  that  can  be  implemented. 

Next,  consider  the  following  probabilities  for  engagement  with  a  single  attacker  shown 
in  Table  6.1  for  four  different  scenarios:  homeland  defense  (HD),  homeland  defense  with  no 
boost-phase  access,  homeland  defense  with  advanced  countermeasures  (CMs),  and  defense  of 
deployed  forces  and  defense  of  friends  and  allies  (DODF/DOFA). 

Assuming  that  the  performances  of  the  systems  in  each  phase  are  independent  of  one 
another,  we  can  compute  the  probability  that  a  single  missile  is  intercepted  for  each  of  the  eight 
investment  options,  as  well  as  for  each  of  the  four  scenarios,  which  can  be  thought  of  as  mea¬ 
sures.  If  we  set  a  threshold  of  0.5  and  a  goal  value  of  0.8  for  each  scenario,  we  get  the  results 
shown  in  Table  6.2  (the  colors  that  would  appear  on  the  Drilldown  sheet)  under  the  default 
(goal-based  with  thresholds)  scoring  method. 

Treating  the  individual  scenarios  as  measures  (or,  to  be  accurate,  treating  each  column  as 
a  measure  consisting  of  a  single  submeasure),  we  can  calculate  the  effectiveness  of  each  invest¬ 
ment  option  for  the  Thresholds  scoring  method.  Assuming  all  scenarios  are  equally  weighted, 
with  a  submeasure  score  of  0.5  for  reaching  the  threshold  value  and  1  for  reaching  the  goal 
value,  the  effectiveness  score  of  each  investment  option  is  as  shown  in  Table  6.3. 

Table  6.1 

Notional  Probabilities  of  Intercept  for  Illustrative  Problem 


Mission  Case 


Flight  Phase 

HD 

No  Boost  Phase 

Advanced  CMs 

DODF/DOFA 

Cost  ($  billions) 

Boost  (B) 

0.7 

0.0 

0.5 

0.7 

9 

Midcourse  (M) 

0.6 

0.6 

0.4 

0.6 

6 

Terminal  (T) 

0.5 

0.5 

0.3 

0.0 

3 

Table  6.2 

Performance  of  Options  (Probabilities  of  Intercept)  by  Mission 


Mission  or  Cost 
Option 

HD 

No  Boost  Phase 

Advanced  CMs 

DODF/DOFA 

Cost  ($  billions) 

B  +  M  +  T 

0.94 

0.8 

0.79 

0.88 

18 

B  +  M 

0.88 

0.6 

0.7 

0.88 

15 

B  +  T 

0.85 

0.5 

0.65 

0.7 

12 

M  +  T 

0.8 

0.8 

0.58 

0.6 

9 

B  only 

0.7 

0.5 

0.7 

9 

M  only 

0.6 

0.6 

0.6 

6 

T  only 

0.5 

0.5 

3 
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Table  6.3 

Costs  and  Effectiveness  Comparisons:  Equal  Emphasis  on  all  Scenarios 


Mission  or 

Cost  Option 

HD 

No  Boost 
Phase 

Advanced 

CMs 

DODF/ 

DO  FA 

Eff. 

(Sum/4) 

Cost 

($  billions) 

B  +  M  +  T 

0.94  [1] 

0.8  [1] 

0.79  [0.98] 

0.88  [1] 

1.0 

18 

B  +  M 

0.88  [1] 

0.6  [0.67] 

0.7  [0.83] 

0.88  [1] 

0.88 

15 

B  +  T 

0.85  [1] 

0.5  [0.5] 

0.65  [0.75] 

0.7  [0.83] 

0.77 

12 

M  +  T 

0.8  [1] 

0.8  [1] 

0.58  [0.64] 

0.6  [0.67] 

0.7 

9 

B  only 

0.7  [0.83] 

0.5  [0.5] 

0.7  [0.83] 

0.54 

9 

M  only 

0.6  [0.67] 

0.6  [0.67] 

0.6  [0.67] 

0.5 

6 

T  only 

0.5  [0.5] 

0.5  [0.5] 

0.25 

3 

None 

0 

0 

NOTE:  The  unbracketed  numbers  are  probabilities  of  successful  intercept,  as  in  Table  6.2.  The  numbers  within 
brackets  are  the  effectiveness  scores  using  the  Thresholds  method. 


Another  possible  perspective  would  be  to  put  a  greater  emphasis  on  dealing  with  a  peer 
threat  that  can  deny  boost-phase  access  and  implement  advanced  countermeasures.  We  can 
examine  that  case  by  weighting  those  two  scenarios  twice  as  much  as  the  other  two.  The  effec¬ 
tiveness  of  each  investment  option  is  then  as  shown  in  Table  6.4. 

The  Scatter  Plot  sheet  can  display  the  effectiveness  of  each  investment  case  for  both  per¬ 
spectives,  with  the  cost  of  the  investment  option  as  the  x-axis  (Figure  6.1). 

The  scatter  plot  view  permits  straightforward  comparison  of  the  investment  options 
under  both  perspectives.  Reading  the  scatter  plot,  however,  requires  some  instruction,  because 
it  is  unconventional.  Suppose  that  the  budget  must  be  cut  from  $18  billion  to  $9  billion. 
At  that  budget  level  (x-axis),  we  see  that  the  best  option  by  the  first  perspective  (squares)  is 
M  +  T  (elimination  of  boost  phase).  That  is,  the  topmost  square  has  the  color  associated  with 
the  M  +  T  option  in  the  legend  box  at  the  right.  When  PAT  is  being  used  live,  the  square  can 
be  identified  by  merely  mousing  over  it  and  seeing  its  name  pop  up. 


Table  6.4 

Costs  and  Effectiveness  Comparisons:  Extra  Emphasis  on  Peer  Threat 


Mission  or 

No  Boost 

Advanced 

DODF/ 

Scaled  Eff. 

Cost 

Cost  Option 

HD 

Phase 

CMs 

DOFA 

(Sum/6) 

($  billions) 

B  +  M  +  T 

0.94  [1] 

0.8  [2] 

0.79  [1.96] 

0.88  [1] 

0.99 

18 

B  +  M 

0.88  [1] 

0.6  [1.33] 

0.7  [1.67] 

0.88  [1] 

0.83 

15 

B  +  T 

0.85  [1] 

0.5  [1] 

0.65  [1.5] 

0.7  [0.83] 

0.72 

12 

M  +  T 

0.8  [1] 

0.8  [2] 

0.58  [1.28] 

0.6  [0.67] 

0.82 

9 

B  only 

0.7  [0.83] 

0.5  [1] 

0.7  [0.83] 

0.44 

9 

M  only 

0.6  [0.67] 

0.6  [1.33] 

0.6  [0.67] 

0.44 

6 

T  only 

0.5  [0.5] 

0.5  [1] 

0.25 

3 

None 

0 

0 

NOTE:  The  unbracketed  numbers  are  probabilities  of  successful  intercept,  as  in  Table  6.2.  The  numbers  within 
brackets  are  the  effectiveness  scores  using  the  Thresholds  method. 
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Figure  6.1 

Cost-Effectiveness  Comparisons  for  Two  Perspectives 


Cost  (SB) 


But  what  if  we  are  more  concerned  about  the  peer  threat  to  the  homeland?  Here,  the 
circles  indicate  the  effectiveness  of  each  investment  option,  and  in  this  case,  the  same  invest¬ 
ment  option  is  best.  That  certainly  might  not  have  been  the  case,  as  can  be  seen,  for  example, 
by  reading  the  chart  for  budget  cuts  of  $3  billion,  $6  billion,  and  $9  billion  and  then  compar¬ 
ing  the  results  to  what  follows. 

In  this  contrived  example,  the  terminal-phase,  midcourse-phase,  and  boost-phase  options 
should  be  canceled  for  the  budget  cuts  of  $3  billion,  $6  billion,  and  $9  billion,  respectively,  to 
maintain  the  greatest  effectiveness  for  the  equal-weighting  perspective.  Although  the  boost- 
phase  option  costs  as  much  as  retaining  the  midcourse-phase  and  terminal-phase  options,  the 
capability  is  much  lower.  This  is  particularly  amplified  under  the  peer-emphasis  perspective 
(the  squares)  when  the  effectiveness  score  of  the  midcourse-only  and  the  boost-phase-only 
options  are  nearly  equivalent.  For  budget  cuts  between  $3  billion  and  $9  billion,  cancellation 
of  the  boost-phase  option  has  greater  effectiveness  than  cancellation  of  the  less-expensive  mid¬ 
course-phase  option  under  the  peer-emphasis  perspective.  Thus,  depending  on  the  perspective 
chosen,  different  priorities  are  placed  on  the  retention  of  different  phases  of  the  defense. 


CHAPTER  SEVEN 


Concluding  Observations 


Purpose  and  Function  of  PAT 

As  described,  from  the  outset,  PAT  is  an  “empty-vessel”  tool,  not  a  model.  Its  purpose  is  to  help 
frame,  manipulate,  analyze,  and  present  results  of  multifaceted  information  to  decisionmak¬ 
ers,  particularly  those  concerned  with  strategic-level  planning.  For  defense,  this  would  include 
what  are  sometimes  called  force  planning,  mission-level  capabilities  planning,  and  cross¬ 
capability-area  planning.  PAT  is  designed  to  work  in  parallel  with  appropriate  capabilities 
models1  and  to  make  use  of  diverse  other  information,  such  as  structured  expert  judgment. 
PAT  can  accommodate  very  different  management  purposes,  including  highlighting  prob¬ 
lems  to  be  confronted,  emphasizing  accomplishments  rather  than  residual  shortcomings,  and 
making  the  relatively  soft  and  forgiving  assessments  that  are  common  in  broad  balance-of- 
power  studies. 


Seeking  Flexibility,  Adaptiveness,  and  Robustness 

Some  who  hear  of  PAT  for  the  first  time  may  think  of  using  it  to  optimize  resource  allocation 
mathematically.  They  may  see  the  mechanism  for  calculating  cost-effectiveness  and  assume 
that  the  objective  is — or  should  be — to  maximize  that  quantity.  That  would  be  a  misreading 
of  our  intentions,  and  indeed  of  the  philosophy  underlying  our  approach.  The  most  important 
outputs  of  PAT  are  (1)  the  portfolio-style  scorecards  in  which  alternative  investment  options 
are  assessed  simultaneously  by  a  number  of  very  different  measures  and  costs  and  (2)  the  next 
layer  of  scorecard  detail  to  which  the  viewer  is  able  to  drill  down  to  understand  the  basis  of 
the  color-coded  summary  assessments  and  to  change  higher-level  assumptions  or  priorities  that 
affect  those  assessments. 

Further  aggregation  to  a  single  number,  as  in  cost-effectiveness  calculations,  should  be 
deferred  to  a  kind  of  refinement  stage,  a  stage  in  which  one  is  “tidying”  and  thinking  about 
communicating  the  results  of  decisions.  We  recommend  this  because  the  cost-effectiveness 


1  A  “capabilities  model”  is  a  relatively  aggregate-level  depiction  of  capability  that  has  attributes  such  as  comprehensibility 
and  parameterization  that  permit  exploratory  analysis  under  uncertainty.  Such  models  differ  from,  e.g.,  the  high-resolution, 
high-fidelity  simulation  models  used  for  training  and  mission  rehearsal.  In  some  cases,  they  can  be  “formula  models”  or 
simple  programs  suitable  for  a  single  analyst  to  use  on  his  personal  computer.  In  other  cases,  e.g.,  the  more  agile  and  com¬ 
prehensible  campaign  models  used  by  DoD  for  defense  planning,  they  are  somewhat  more  complicated.  Thunder,  STORM, 
and  JICM  are  capability  models  of  this  type.  JWARS  is  not,  nor  are  models  such  as  Brawler.  Within  the  emerging  realm  of 
models  for  irregular  warfare,  the  often-mentioned  models  (e.g.,  PSOM,  SEAS,  COMPOEX)  are  much  more  complicated, 
although  they  have  modules  that  are  analogous  to  capability  models. 
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calculations  depend  sensitively  on  the  assumptions  and  priorities  that  go  into  them,  which  are 
precisely  what  decisionmakers  are  paid  to  think  about  and  decide  on.  The  decisionmakers  are 
responsible  for  worrying  about,  say,  the  “balance”  of  a  portfolio  across  missions,  the  extent  of 
risk  to  be  taken,  and  the  ways  in  which  risk  can  be  managed.  Therefore,  they  need  to  reason  at 
the  portfolio  level,  not  at  the  level  where  they  are  merely  comparing  cost-effectiveness  numbers. 

PAT  provides  a  number  of  ways  to  assess  alternatives  and  some  useful,  albeit  limited, 
mechanisms  for  exploring  the  consequences  of  alternative  assumptions  and  priorities,  but  that 
is  very  different  from  optimization.  The  purpose  is  to  find  strategies  that  enjoy  “FARness” 
(flexibility,  adaptiveness,  and  robustness)  within  plausible  budgets.2 

PAT  as  Software 

PAT  is  not  industrial-strength  software;  rather,  it  is  a  tool  for  relatively  sophisticated  analysts 
who  worry  more  about  functionality  than  about  cosmetics.  It  has  been  tested  in  a  number  of 
applications,  but  it  undoubtedly  has  residual  problems.  Users  are  encouraged  to  contact  PAT’s 
developer,  Paul  Dreyer,  at  dreyer@rand.org  if  they  discover  mistakes  or  have  technical  ques¬ 
tions  about  PAT. 

PAT  checks  for  most  mistakes  that  users  make  (for  example,  entering  an  investment 
option  or  measure  in  the  Summary  table  that  does  not  exist  on  a  data  sheet).  However,  PAT 
has  not  been  exhaustively  tested  or  “gorilla-proofed,”  nor  has  it  been  refined  in  the  manner  of 
commercial  software.  Neither  the  authors  nor  RAND  offers  any  guarantees  or  warranties  on 
its  use.  We  encourage  users  to  keep  a  clean  copy  of  the  PAT  template  available  in  the  unlikely  event 
that  something  occurs  to  make  the  software  unusable. 

The  proper  use  of  PAT  also  requires  discipline.  We  have  attempted  to  simplify  much  of 
the  initial  setup  of  a  portfolio  view  in  the  Template  Builder  sheet,  but  other  operations  require 
consistency.  For  example,  if  one  edits  the  values  on  a  Drilldown  sheet,  it  is  necessary  to  click 
on  the  Modify  Data  Entries  button  to  make  those  changes  propagate  through  the  rest  of  the 
tool.  Similarly,  unless  the  Auto-Update  Chart  checkbox  is  checked  on  each  sheet,  the  Scatter 
Plot  sheet  and  the  Spider  Chart  sheet  do  not  automatically  update  to  reflect  changes  that  have 
been  made  to  the  data  sheets;  it  is  necessary  to  click  on  the  Generate  Scatter  Plot  and  Generate 
Spider  Chart  buttons  on  the  applicable  sheets  to  see  how  the  changes  in  the  data  have  affected 
the  outputs.3 

PAT  allows  considerable  flexibility  in  what  can  be  changed  without  having  to  regenerate 
the  portfolio  view,  as  described  above.  Weights  of  measures  and  submeasures,  threshold  and 
goal  values,  scoring  methods,  and  data  values  can  be  changed  easily. 

The  only  constraint  on  the  numbers  of  investment  options,  measures  and  submeasures, 
etc.,  that  can  be  used  is  the  ability  of  a  spreadsheet  to  hold  all  of  the  input  data.  That  should 
not  be  constraining  in  practice. 

The  Importance  of  the  Measures  and  Methods 

As  with  all  scorecard  methods,  considerable  care  must  be  taken  in  the  development  of  the  mea¬ 
sure/submeasure  structure  for  any  analysis  performed  with  PAT.  Similar  care  should  be  taken 
with  the  selection  of  the  scoring  method  and  the  parameters  and  weights  for  the  scoring  func- 


2  This  emphasis  in  RAND  work  has  been  articulated  in  a  number  of  RAND  monographs  (Davis,  1994;  Davis,  2002a).  It 
was  also  highlighted  in  a  recent  National  Academy  study  (National  Research  Council,  2006). 

3  As  PAT  is  continuously  improved,  more  automatic  updating  is  being  introduced. 
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tions  and  effectiveness  calculations.  If  PAT  used  only  linear  weighted  sums,  the  methodology 
would  be  simpler  but  less  satisfactory  for  system  analysis. 

Although  we  cannot  itemize  here  all  the  considerations  that  an  analyst  should  have  in 
mind,  a  few  are  particularly  worth  mentioning,  if  only  as  a  partial  checklist: 

•  Measures  and  submeasures  should  provide  an  adequately  complete  assessment. 

•  Ideally,  measures  would  be  independent,  and  the  submeasures  of  each  measure  would 
be  independent.  When  that  is  not  appropriate — i.e.,  when  correlations  exist — weighting 
factors  should  be  chosen  so  as  to  avoid  results  being  overly  sensitive  to  a  single  underlying 
issue.4 

•  The  choice  of  scoring  and  aggregation  methods  merits  particular  thought.  If  all  of  a  mea¬ 
sure’s  submeasures  are  individually  critical,  then  the  Threshold  method  may  be  called  for. 
If  all  of  the  measures  are  also  individually  critical,  then  the  Weakest  Link  Method  may 
be  appropriate.  If  these  conditions  do  not  apply,  however,  and  one  is  more  interested  in 
seeing  progress  than  in  flagging  problems,  then  the  Goals  method  may  be  appropriate. 
The  Rankings  method  can  be  helpful  when,  for  one  reason  or  another,  it  is  inappropriate 
to  discuss  goals  and  thresholds. 

•  In  any  case,  it  is  essential  to  plan  for  systematic  exploration  of  how  assumptions  on  the 
above  matters  affect  both  results  and  perceptions,  and  to  tune  assumptions  so  as  to  pro¬ 
vide  a  set  of  baseline  results  that  are  as  robust  as  possible.  For  example,  it  serves  decision¬ 
makers  poorly  when  color-coded  conclusions  change  markedly  if  some  low-level  assump¬ 
tions  are  changed  slightly  (e.g.,  moving  a  goal  from  0.89  to  0.9  should  not  change  results 
dramatically). 

•  A  consequence  of  the  above  admonition  is  that  goals  and  thresholds  need  to  be  seen  as 
heuristics,  not  as  absolutes  to  be  accepted  mindlessly. 

Finally,  we  note  that  many  of  these  issues  are  generic.  There  is  a  considerable  literature 
dealing  with  multiattribute  measures  and  objectives  that  discusses  approaches  to  weighting 
these  measures,  obtaining  utilities  from  individuals  or  groups,  and  the  use  of  other  aggregation 
rules.5 


Next  Steps 

Over  time,  PAT  will  be  improved  and  enhanced,  building  on  the  experience  of  applica¬ 
tions.  Suggestions  will  be  appreciated.  In  addition  to  correcting  errors  and  improving  user- 
friendliness,  we  are  currently  thinking  about  at  least  the  following  possibilities  for  enhancement: 

•  Permitting  different  scoring  and  aggregation  methods  to  be  used  for  different  measures  or 
for  calculation  of  cost-effectiveness  rather  than  measure  scores. 


4  For  example,  an  option  could  be  made  to  look  better  by  piling  on  a  number  of  measures,  each  of  which  is  driven  by 
something  accomplished  well  by  the  option.  Similarly,  an  option  may  appear  worse  than  it  probably  should  if  the  measures 
chosen  reflect  a  pure  worst-case  perspective. 

5  Some  of  these  approaches  appear  under  discussions  of  multiattribute  utility  theory  (Keeney  and  Raifa,  1976;  Kirkwood, 
1997),  value-focused  thinking  (Keeney,  1992;  Parnell,  2006),  and  balanced-scorecard  methods  used  in  business  (Kaplan 
and  Norton,  1996).  The  original  DynaRank  documentation  also  includes  some  discussion  of  these  approaches  (Hillestad 
and  Davis,  1998  ). 
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•  Providing  the  ability  to  generate  the  measure  of  cost  used  in  cost-effectiveness  as  a  linear 
weighted  sum  of  the  various  costs  provided  as  inputs. 

•  Developing  a  richer  and  more  structured  mechanism  for  exploratory  analysis,  probably 
building  on  the  alternative-perspectives  mechanism.  This  could  include  limited  mecha¬ 
nisms  for  search  (e.g.,  finding  the  combinations  of  key  parameters  that  would  cause  a 
particular  option  to  be  assessed  well  or  poorly). 


APPENDIX  A 


Quickstart  on  Using  PAT 


This  appendix  is  written  for  those  who  want  to  learn  by  doing,  at  least  initially.  It  assumes  that 
the  reader  has  a  copy  of  the  relevant  Excel  file  for  PAT.  The  format  is  informal.  The  example  is 
the  one  used  in  the  main  text. 


Opening  PAT 

When  you  open  PAT,  you  may  be  asked  whether  to  enable  macros.  If  you  are  asked,  the  answer 
is  Yes.  If  the  computer  refuses  to  open  the  file  for  security  reasons,  you  will  need  to  reset  the 
level  of  protection: 

•  Close  and  reopen  Excel 

•  Go  to  the  top-of-window  menu  Tools/Macro/Security 

•  Set  security  level  to  “medium.” 

You  will  now  be  permitted  to  continue.1 


Navigation  and  Manipulation 

To  begin,  PAT  will  open  a  Summary  sheet  (Figure  A.l)  with  some  placeholder  data,  which  you 
will  replace.  Although  you  will  seldom  work  with  the  entire  Summary  sheet  at  a  given  time, 
you  should  be  aware  that — as  shown  in  the  figure — it  has  four  different  vertical  blocks  of 
information.  The  leftmost  block  is  the  scorecard  that  characterizes  options  by  different  mea¬ 
sures  of  effectiveness  (measures).  The  second  block  is  a  set  of  columns  that  allow  the  user  to 
show  selected  data,  by  option,  from  the  second  level  of  detail.  The  third  block  contains  selected 
output  cost  information.  The  final  (rightmost)  block  contains  calculations  of  the  options’  effec¬ 
tiveness  and  cost-effectiveness  when  aggregated  across  all  of  the  measures.  Ordinarily,  the  user 
will  focus  on  the  scorecard  portion  of  the  Summary  sheet. 


1  This  instruction  is  for  users  of  Excel  in  Microsoft  Office  2003  for  Windows  XP.  You  may  not  encounter  any  such  warning 
if  you  are  using  Excel  2004  for  the  Macintosh.  The  procedure  for  adjusting  the  security  settings  is  slightly  different  in  Excel 
2007  because  of  changes  in  the  interface. 
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Figure  A.1 

Overview  of  Summary  Sheet 


Options  Menu 


Sorting  Category 


Multi-Resolution  Modeling  Level 


Please  Select  An  Option 

M 

List  of  Sorting  Categories 

|  ▼  |  Use  Level  3  Data  |  ▼  | 
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Cost  Effectiveness  Cost  Metric 
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M 

Largest  Value  First 

▼  Total  costs:  2010-2015  ($M)  | 

Current  Perspective 
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FI 

0  M 

If  you  are  looking  at  the  Summary  sheet  on  a  computer,  note  along  the  bottom  of  the 
window  that  the  Summary  tab  is  highlighted.  You  can  switch  among  sheets  by  clicking  other 
tabs.  Since  there  are  many  such  tabs,  it  is  often  easier  to  use  the  Go  To  Sheet  menu  item  at  the 
top  of  your  screen.  Custom  sheets  will  not  be  listed  (e.g.,  sheets  for  notes  or  for  specialized  cost 
calculations  not  done  by  PAT,  or  a  miniature  capabilities  model  that  can  generate  data  entries 
for  PAT’s  primary  sheets). 

On  the  Summary  sheet,  mousing  over  (dragging  the  cursor  over)  the  various  menus’ 
arrows  reveals  the  following: 


•  The  Options  menu  contains  miscellaneous  controls,  such  as  whether  to  show  relative 
weights  and  whether  to  change  coloring  scheme.  You  can  select  the  change  in  coloring 
scheme  to  see  what  happens;  repeat  four  times  to  get  back  to  where  you  started. 

•  Scoring  Method  allows  you  to  select  a  scoring  method,  as  discussed  in  the  text.  Thresh¬ 
olds  is  the  default  method.  That  is  equivalent  to  linear  weighted  sums  if  the  thresholds 
and  goals  are  set  to  0  and  1,  respectively.  An  important  alternative  is  the  Weakest  Link 
method,  i.e.,  when  a  score  should  be  no  better  than  the  weakest  of  its  contributors. 

•  The  Current  Perspective  menu  allows  you  to  choose  from  among  various  perspectives  that 
you  have  previously  defined  or  to  make  new  ones.  A  perspective  is  characterized  by  its 
weighting  schemes  and  other  parameters.  There  are  none  at  the  outset. 

•  Sorting  Category  and  Sorting  Method  allow  you  to  select  a  column  and  then  sort  rows 
based  on  some  criterion.  This  is  discussed  in  Dreyer  and  Davis  (2005),  although  some 
details  have  changed. 

•  Multi-Resolution  Modeling  Level  gives  you  a  choice  of  which  of  three  databases  to  use. 
The  option  of  entering  information  at  a  higher  level  is  part  of  a  multiresolution  model¬ 
ing  philosophy.  Use  Level  3  Data  actually  means  enter  data  at  Level  3  and  Level  2.  Use 
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MRM  Level  2  Data  means  enter  data  only  at  Level  2.  Use  MRM  Level  1  Data  means 
enter  data  only  at  Level  1,  essentially  specifying  answers  as  you  might  when  experiment¬ 
ing  with  story  line  and  displays.  If  you  want  to  do  a  first,  rough-cut  analysis,  start  with 
Level  2. 

•  Cost  Effectiveness  Cost  Metric  is  self-evident,  referring  to  the  cost  that  is  to  be  used  when 
evaluating  relative  cost-effectiveness  by  dividing  effectiveness  by  a  cost.  The  options  on 
the  menu  depend  on  what  you  specify  for  costing  options  in  the  Template  Builder  or  the 
Cost  Data  sheet,  both  of  which  are  described  later. 

•  Discount  Rate  gives  you  the  option  of  doing  present-value  calculations  by  selecting  the 
real  discount  rate,  assuming  that  the  options’  costs  are  already  in  constant  dollars.  If  they 
are  in  current  dollars,  you  would  use  a  discount  rate  equal  to  inflation  plus  the  real  dis¬ 
count  rate. 

The  illustrative  Summary  sheet’s  first  column  should  be  read  as  Investment  Options.  The  word 
“Measures”  pertains  to  the  subsequent  columns  in  row  1.  The  phrase  “Level  1”  is  present  tem¬ 
porarily  and  is  later  replaced  by  the  name  of  the  first  Level  1  measure. 

Subsequent  columns  that  contain  Detail  buttons  are  reserved  for  potential  measures  of 
the  options’  goodness  or  utility.  As  suggested  by  Figure  A.l,  if  you  scroll  to  the  right  you  will 
encounter  a  block  of  columns  reserved  for  cost  information.  Which  cost  columns  appear  here 
depends  on  information  entered  elsewhere.  You  might  have  columns  for,  say,  R&D,  acquisi¬ 
tion,  and  O&S,  as  well  as  total  costs  in  a  particular  period,  etc. 

Continue  scrolling  and  you  will  find  columns  to  the  right  that  are  reserved  for  net  effec¬ 
tiveness  and  relative  cost-effectiveness.  Those  show  up  only  after  relevant  options  and  data  have 
been  entered. 

The  Detail  buttons  allow  you  to  drill  down  to  a  second  level  of  detail,  one  that  “explains” 
the  summary-level  results  for  the  column  in  question. 


Template  Builder 

Template  Builder's  Structure 

Go  next  to  the  Template  Builder  sheet  (at  the  far  right  of  the  tabs  or  via  Go  To  Sheet/Inputs). 
This  is  where  you  enter  the  information  that  dictates  the  overall  structure  of  your  portfolio 
analysis.  Figure  A.2  shows  Template  Builders  structure.  The  various  blocks  in  dashed-edge 
red  rectangles  are  independent.  That  is,  do  not  assume  that  two  items  relate  to  each  other  just 
because  they  are  in  adjacent  columns.  Items  in  a  given  block  relate  to  each  other.  This  is  what 
you  have  to  do  for  each  block  (the  red  text  items  are  placeholders): 

•  Fill  in  the  names  of  your  options,  one  line  per  option.  “Baseline”  is  the  placeholder 
example. 

•  Indicate  the  range  of  years  that  you  wish  to  consider  (2010-2030  in  the  example). 

•  Define  the  units  of  currency  (“Thousands”  in  the  example,  although  that  surely  doesn’t 
apply  to  government). 

•  Enter  the  names  of  Level  1,  Level  2,  and  Level  3  measures,  along  with  some  control 
parameters  for  the  latter.  This  is  a  large  and  relatively  complicated  block. 


Figure  A. 2 

Illustrative  Template  Builder  Sheet 
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•  List  the  items  for  which  you  want  to  track  costs  separately  (Item  1  in  the  example). 
This  could  be,  e.g.,  ships,  airplanes,  .  .  .  ;  or  radars,  interceptors,  .  .  .  ;  or  human  capital, 
equipment,  ....  A  word  of  caution:  If  you  have  numerous  items  here,  the  data  require¬ 
ments  quickly  become  very  burdensome.  If  you  don’t  want  to  bother,  you  might  replace 
“Item  1”  with  “Stuff”  or  “Miscellaneous.” 

•  List  the  investment  “categories”  for  which  you  want  to  track  costs  separately  (“Main”  is 
the  placeholder).  The  usual  choices  here  might  be  “Total”  (meaning  don’t  bother  with 
categories)  or,  e.g.,  the  items  “R&D,”  “acquisition,”  and  “O&S.” 

Blocks  A,  B,  and  C  require  little  explanation,  but  let  us  elaborate  on  Block  D.  The  syntax 
for  entering  data  in  the  first  three  columns  is  shown  in  Table  A.l.  We  assume  here  that  you 
have  three  measures,  as  in  the  text,  with  only  the  second  measure  having  third-level  data. 
Measure  1.1  appears  to  the  immediate  right  of  Measure  1,  and  Measure  2.2.1  appears  to  the 
immediate  right  of  Measure  2.2.  The  name  of  a  measure  should  not  be  repeated  if  the  next  row 
is  simply  filling  in  a  submeasure. 

You,  of  course,  will  want  to  have  more  interesting  names  for  your  various  measures  and 
submeasures.  The  following  are  some  examples  of  the  kinds  of  measures  you  might  use  at  the 
first,  second,  and  third  levels  of  detail  (separated  by  semicolons):2 

Force  Structure:  Regions  of  the  World.  Within  each,  environment-shaping  versus  war¬ 
fighting;  within  warfighting,  expectations  for  different  test  cases.  Additional  measures  could 
be  added  for  risks  of  various  types  at  each  level  of  analysis. 

Acquisition  of  Weapon  Systems.  Broad  mission  classes;  within  each,  suitable  test  sce¬ 
narios;  within  each,  suitable  technical  measures  (e.g.,  probability  of  being  able  to  penetrate  air 
defenses). 

International  Business.  Operating  divisions;  measures  of  performance  in  each  operating 
division;  diagnostic  measures  to  explain  performance. 

Planning  of  Staff  for  a  Knowledge-Industry  Business.  Business  areas;  within  each, 
core  expertise  (leadership),  quality,  number,  diversity,  and  affordability  of  staff  mix;  within,  say, 
the  measure  of  quality,  various  diagnostics  such  as  experience,  highest  degree  (or  equivalent), 
specialty  experience  in  the  business  area,  and  past  performance. 


Table  A.l 

Format  for  Entering  Measure  Names  in  Template  Builder 


Measure  1 

Measure  1.1 

Measure  1.2 

Measure  2 

Measure  2.1 

Measure  2.2 

Measure  2.2.1 

Measure  2.2.2 

Measure  3 

Measure  3.1 

Measure  3.2 

2  For  some  actual  examples  from  past  studies,  see  Davis,  Kugler,  and  Hillestad  (1997),  Davis,  Shaver,  and  Beck  (2008),  and 
Davis  et  al.  (2008). 
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None  of  the  above  is  to  be  taken  literally.  The  examples  are  merely  starting  points. 

If  you  click  on  the  Show/Hide  Example  button,  an  example  will  pop  up  with  entries 
very  similar  to  those  used  for  the  example  in  the  main  text.  Click  the  button  again  to  hide  the 
example. 

Reviewing  Your  Template  Builder  Sheet 

Review  your  sheet  carefully  to  make  sure  that  there  are  no  inappropriate  blanks  and  that 
syntax  is  exactly  as  in  the  examples.  This  is  where  most  errors  occur  in  building  PAT  work¬ 
sheets.  For  example,  the  word  Threshold  may  not  appear  in  the  last  row  listing  the  measures 
and  submeasures  because  you  added  one  and  forgot  to  fill  in  that  column.  If  all  looks  well,  click 
Build  Sheets  (top  left  corner). 

You  will  be  prompted  about  whether  you  want  to  save  data.  If  you  are  starting  fresh,  the 
answer  is  No,  but  if  you  are  rerunning  Template  Builder  to  reflect  some  changes  in  the  middle 
of  a  project,  you  may  want  to  save  the  data.  In  fact,  you  will  be  saving  only  the  data  associated 
with  options  and  measures  that  still  have  the  same  names  in  the  new  structure,  but  that  may 
be  quite  a  lot. 

Once  Template  Builder  has  stopped  running,  go  to  the  Summary  sheet.  Select  Update 
Summary  from  the  Options  menu,  and  the  Summary  will  be  generated.  You  should  then  see 
the  desired  structure  of  headers.  Check  carefully;  some  error  may  have  occurred,  in  which  case 
you  may  not  see  all  the  option  names  or  all  the  top-level  measures.  Click  on  the  Detail  buttons 
to  see  whether  the  Level  2  structures  are  correct.  Then  click  on  the  Level  3  Drilldown  tab  to 
see  whether  the  appropriate  Level  3  measures  (if  any)  are  available  from  the  menu  box  at  the 
top  left  (Level  3  Drilldown  Options). 


Data  Entry 

Chapter  Three  presents  completed  examples  of  all  the  data-entry  sheets,  so  we  will  be  brief 
here. 

Let  us  assume  that  you  want  to  use  Level  2  and  Level  3  data,  rather  than  the  MRM 
options,  in  which  you  enter  data  only  at  Level  2  or  Level  1 .  Go  to  the  Level  3  Data  sheet  (via 
the  tab  or  the  Go  To  Sheet  menu).  (See  Chapter  Three  for  a  filled-out  example.) 

Enter  weights  for  the  different  Level  3  measures  in  the  fourth  row.  These  might  be,  e.g., 
4  and  2,  in  which  case  PAT  will  normalize  them  as  necessary.  Instead,  you  could  have  entered 
2/3  and  1/3.  For  the  example,  however,  you  can  use  1  and  1. 

Rows  five  through  eight  tell  PAT  how  to  handle  the  data.  Since  we  want  high  values,  we 
might  set  the  values  at  0  and  10  for  rows  five  and  six — meaning  that  the  data  entered  will  be 
between  0  and  10,  inclusive.  However,  you  will  then  want  0  and  1  in  rows  nine  and  ten,  so  that 
PAT  will  rescale  to  that  range.  If  you  had  specified  that  “good”  was  “low,”  you  would  reverse 
the  values  in  rows  five  and  six.  Had  you  chosen  something  other  than  Thresholds,  the  proce¬ 
dure  might  be  slightly  different. 

Now  enter  data  in  the  0-to-10  range  for  your  options. 

When  you  are  done,  click  on  Modify  Level  2  Data  (top  left).  Select  Yes  about  updating. 

You  will  now  be  taken  to  the  Level  2  Data  sheet,  where  you  should  fill  in  data  in  the  same 
way;  some  data  (in  italics)  will  already  be  filled  in,  the  result  of  the  Level  3  calculation.  Do  not 
type  in  these  cells,  obviously,  or  you  will  be  overwriting  the  calculations.  You  might  want  to 
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do  that  for  some  reason,  but  remember  that  any  of  a  number  of  operations  with  PAT  will  cause 
recalculation,  in  which  case  your  overwrites  will  be  overwritten!  If  you  really  want  to  “think” 
at  Level  2,  you  should  be  using  the  optional  Level  2  MRM  approach. 

Note:  Enter  0  and  1  as  the  values  for  Threshold  Value  and  Goal  for  the  columns  of  values 
calculated  from  Level  3,  since  PAT  has  generated  numbers  assuming  this  result. 

When  you  are  done  entering  data,  click  Modify  Summary  (top  left).  You  will  be  taken  to 
the  Summary  sheet,  where  you  should  see  the  colorful  scorecard  display. 


Costs 

Next,  you  need  to  enter  costs  for  the  options.  Do  this  in  the  Cost  Data  sheet.  It  will  have  a 
structure  corresponding  to  the  number  of  years  that  you  specified  in  Template  Builder.  You 
need  to  enter  costs  by  year,  by  option,  for  each  category  of  investment,  and  for  each  type  of 
investment  item,  which  may  simply  be  “none”  as  in  the  example.  To  save  trouble  at  this  point, 
you  can  enter  all  the  costs  in  the  cells  for  2010.  Or  you  could  go  back  to  Template  Builder  and 
start  over  again  with  only  a  single  category,  “Total,”  and  a  single  item,  “Stuff.” 

The  default  is  that  the  costs  are  in  $  millions.  Once  you’re  done,  click  Modify  Summary. 


Cost-Effectiveness 

You  will  be  taken  back  to  the  Summary  sheet.  If  you  scroll  far  enough  to  the  right,  you  will  see 
columns  with  the  costs,  an  effectiveness  measure,  and  relative  cost-effectiveness.  The  effective¬ 
ness  is  merely  a  linear  weighted  sum  over  the  various  measures,  using  weights  that  you  speci¬ 
fied  in  the  Level  2  data.  To  see  those  weights,  toggle  the  corresponding  control  in  the  Options 
menu.  You  can  actually  change  the  weights  here  as  well,  although  their  proper  place  for  change 
is  in  the  Level  2  Data  sheet. 


Scatter  Plot 

PAT  has  many  displays,  one  of  which  is  the  scatter  plot.  To  access  it,  go  to  the  Scatter  Plot  tab. 
It  will  probably  be  blank.  If  it  is,  click  on  Generate  Scatterplot.  You  will  now  see  each  option 
represented  by  a  dot  at  a  point  in  the  scatter  plot  corresponding  to  its  effectiveness  and  cost. 
Such  cost-effectiveness  landscapes  are  good  for  understanding  cost  and  effectiveness  together. 
They  are  much  better  than  working  only  with  the  ratio  of  effectiveness/cost.  However,  remem¬ 
ber  that  the  effectiveness  calculation  depends  on  the  weights  given  to  the  measures  and — in 
this  simple  example — on  the  assumption  of  linear  weighted  sums. 


An  Exercise 

To  conclude  this  Quick-Start  material,  we  provide  an  exercise  for  you. 

In  this  exercise,  you  want  to  use  PAT  to  compare  a  baseline  option  (Option  1)  with  four 
alternatives  (2,  3,  4,  and  5).  Suppose  you  want  to  assess  them  in  terms  of  their  effectiveness 
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in  two  scenarios,  A  and  B,  and  a  catch-all  category  of  “Other”  (in  national-security  work  this 
might  include  something  like  “Shaping  the  Environment”).  In  characterizing  effectiveness, 
you  distinguish  between  results  using  “best-estimate”  assumptions,  best-case  assumptions  (a 
measure  of  upside  potential),  and  worst-case  assumptions  (a  measure  of  risk).  In  evaluating  the 
worst-case  effectiveness  for  Scenario  B,  you  feel  that  it  is  necessary  to  consider  several  “bad” 
variants  of  Scenario  B  which  stress  capabilities  in  different  ways.  They  are  called  variants  B.l, 
B.2,  and  B.3. 

Assuming  that  all  these  measures  have  been  appropriately  defined  and  put  on  the 
same  scale  (from  0  to  1),  you  might  have  the  following  data  sets,  where  larger  numbers  are 
always  better  for  the  measures  and  where  you  need  not  worry  about  subtleties  such  as  thresh¬ 
olds  and  ceilings.  The  data  in  Tables  A.2  through  A.4  may  be  based  on  studies  that  used 
organization- approved  models  and  data  sets  for  the  scenarios.  Some  translation  must  have  been 
made  between  outputs  of  models  and  the  scores  shown  for  the  options,  but  we  will  not  concern 
ourselves  about  that  here. 


Table  A.2 

Level  2  Data  for  Illustrative  Exercise 


Option 

Best-Estimate  Case  Best  Case 

Worst  Case 

Scenario  A 

1  (Base) 

0.1 

0.1 

0.1 

2 

0.5 

0.6 

0.3 

3 

0.55 

0.7 

0.5 

4 

0.57 

0.9 

0.5 

5 

1 

1 

0.5 

Scenario  B 

1  (Base) 

0.1 

0.1 

Calculated 

2 

0.41 

0.55 

Calculated 

3 

0.45 

0.7 

Calculated 

4 

0.6 

0.82 

Calculated 

5 

0.75 

1 

Calculated 

Other  Measures 

Level  2  Score 

1  (Base) 

0.1 

2 

0.5 

3 

0.5 

4 

0.5 

5 

1.0 
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Table  A. 3 

Level  3  Data  for  Illustrative  Exercise 


Outcomes  for  Different  Definitions  of  Worst  Case 

Option 

B.1 

B.2 

B.3 

1  (Base) 

0.1 

0.1 

0.1 

2 

0.1 

0.3 

0.3 

3 

0.1 

0.3 

0.3 

4 

0.3 

0.5 

0.3 

5 

0.5 

0.5 

0.3 

Table  A. 4 

Cost  Data  for  Illustrative  Exercise 


Option 

Cost  ($  millions) 

Baseline  (Option  1) 

0 

2 

10 

3 

80 

4 

160 

5 

200 

Now  suppose  that,  as  a  first  approximation,  you  think  of  overall  effectiveness  as  an  aver¬ 
age  of  that  for  Scenarios  A  and  B  and  as  an  average  of  results  for  best-estimate,  best-case,  and 
worst-case  variants.  Indeed,  even  in  thinking  about  the  worst-case  outcome  for  Scenario  B, 
which  has  troubling  variants,  you  take  an  average. 

With  this  background,  the  exercise  is  as  follows: 

•  Starting  with  Template  Builder,  set  up  PAT  appropriately  and  enter  data,  recognizing  that 
the  data  provided  here  are  not  in  the  proper  format  for  PAT.  That  is,  you  will  have  to 
translate  these  data  into  PAT’s  terms. 

•  Specify  weights,  thresholds,  and  goals  appropriately. 

•  When  everything  is  working,  look  at  the  various  output  sheets  to  check  for  any  egregious 
mistakes. 

•  Recalling  that  different  types  of  information  are  separated  horizontally  on  the  Summary 
sheet  (see  Figure  A.l),  use  standard  Excel  functionality  to  arrange  the  sheet  so  that  you 
can  see  the  scorecard  and,  immediately  adjacent  to  it,  the  columns  for  cost  and  effective¬ 
ness.  This  will  require  hiding  some  columns  and  creating  a  two-pane  view.  To  hide  col¬ 
umns,  select  them  and  choose  Column/FIide  from  the  Excel  Format  menu.  To  create  the 
two-pane  view  use  the  “tug  bar”  located  at  the  bottom  right-hand  corner  of  the  display 
(see  Figure  A.3).3 

•  Generate  the  scatter  plot  of  effectiveness  versus  cost. 

If  you  have  done  everything  correctly,  the  results  should  look  like  those  shown  in  Figures  A.4 
through  A.7. 


3  Microsoft  documentation  refers  to  “splitting  or  freezing  panes”  when  discussing  these  matters.  Precise  terminology  may 
differ  in  different  versions  of  Excel. 
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Figure  A. 3 

Tug  Bar  for  Viewing  Separated 
Portions  of  an  Excel  Spreadsheet 


Figure  A. 4 

Summary  Sheet  Excerpt  for  Exercise  Problem 


Measures 

Effect,  in 
Scenario  A 

Effect  in 
Scenario  B 

Other 

Measures 

Total  Cost: 
2010-2030 
($M) 

Relative  Cost 
Effectiveness  Effectiveness 

Investment  Options 

Detail 

Detail 

Detail 

Detail 

Detail 

i 

i 

i 
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Figure  A. 5 

Level  2  Drilldown  for  Exercise  Problem 


Effect,  in 
Scenario 

Level  1  Measure 

B 

Best- 

Worst- 

Best- 

Case 

Case 

Estimate 

Outcome 

Outcome 

Level  2  Measure 

Outcome 

(Upside) 

(Risk)  Warning 

Investment  Options 


Baselne  (Option  1) 


Option  2 

Option  3 

Option  4 

Option  5 


Effect,  in 
Scenario 
B  Score 


0.41 


0.45 


0.60 


0.75 


0.55 


0.70 


0.82 


1.00 


0.23 


0.23 


0.37 


0.43 


uepenus 

on 

Technolo 


0.23 


0.23 


0.37 


0.43 


Figure  A. 6 

Level  3  Drilldown  for  Exercise  Problem 


Level  1  Measure 

Effect,  in  Scenario  B 

Level  2  Measure 

Worst-Case  Outcome  (Risk) 

Level  3  Measure 

Scenario  B.1  Scenario  B.2 

Scenario  B.3 

Investment  Options 


Worst-Case 
Outcome  (Risk) 
Score 


Baselne  (Option  1) 


Option  2 

0.30 

0.30 

0.23 

Option  3 

0.30 

0.30 

0.23 

Option  4 

0.30 

0.50 

0.30 

0.37 

Option  5 

0.50 

0.50 

0.30 

0.43 
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Figure  A. 7 

Scatter  Plot  of  Effectiveness  Versus  Cost  for  Exercise  Problem 


Total  costs:  2010-2030  ($M) 


These  figures  are  worth  pondering,  because  they  make  a  point  about  the  relative  useful¬ 
ness  of  different  information.  Figure  A.4  is  a  good  condensation  of  information  for  the  analyst: 
It  shows  a  multiobjective  or  multicriteria  scorecard  comparing  the  options,  and  it  also  shows 
cost  and  aggregate  effectiveness.  In  our  experience,  focusing  too  early  on  the  aggregate  effec¬ 
tiveness  is  a  bad  idea.  Who,  after  all,  really  understands  what  an  effectiveness  of  0.5  rather  than 
0.4  means?  The  scorecards  in  Figures  A.4  through  A. 6  provide  much  more  information  than  a 
single  effectiveness  number — information  needed  for  decisionmaking. 

Effectiveness  is  useful  primarily  for  summary  charts  after  the  decisionmaking  has 
occurred,  perhaps  for  telling  the  story  of  effectiveness  versus  cost,  as  shown  in  Figure  A.7. 
The  “landscape”  of  effectiveness  versus  cost  is  quite  understandable  but  much  more  informa¬ 
tive  than,  say,  the  cost-effectiveness  ratio  at  some  particular  value  of  cost.  In  this  example, 
Option  2  buys  quite  a  lot  for  modest  sums  of  money  ($10  million).  Improving  results  further, 
however,  requires  “buying  in”  to  a  next  big  increment  of  capability.  That  is,  one  must  spend  a 
good  deal  more  ($80  million)  to  achieve  a  sizable  jump  in  effectiveness.  Pushing  effectiveness  to 
its  upper  limits,  the  goal  of  1.0,  is  several  times  more  expensive  yet  ($200  million).  As  shown  in 
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the  edited  version  of  the  chart  (Figure  A.8),  it  may  be  appropriate  to  connect  the  points  for 
greater  visual  clarity.  In  this  case,  we  assumed  that  there  are  no  good  options  between  the 
points  shown,  so  the  dashed  line  constitutes  what  economists  call  the  “efficient  frontier.”  At 
any  given  level  of  cost  (x-axis),  there  is  no  option  with  greater  effectiveness  than  the  line,  and 
for  any  given  level  of  effectiveness  (y-axis),  there  is  no  option  with  lower  cost. 

Figure  A.8 

Annotated  Scatter  Plot 
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APPENDIX  B 


Transferring  Data  from  an  Earlier  Workbook 


Users  working  with  PAT  frequently  want  to  move  at  least  some  data  from  a  previous  workbook 
into  a  new  one.  Perhaps  some  errors  have  been  made  and  results  are  now  confusing.  Perhaps 
one  has  received  an  updated  version  of  PAT  that  corrected  some  bugs.  Or  perhaps  one  just 
wants  to  start  a  number  of  things  fresh.  The  recommended  procedure  for  copying  into  a  new 
copy  of  PAT  (a  copy  of  “Clean  PAT”)  is  as  follows: 

•  Assuming  that  the  old  Template  Builder  is  correct,  copy  and  paste  data  from  it  first.  That 
is,  copy  and  paste  the  rows  for  the  investment  options  and  their  measures  (rows  3  .  .  .  ). 
Then  enter  or  manually  set  the  few  remaining  items,  such  as  the  range  of  years  intended 
and  the  cost  units  (from  the  related  menu).  Alternatively,  you  can  copy  and  paste  the 
entire  sheet. 

•  Run  Template  Builder  in  the  new  sheet  (i.e.,  click  on  Build  Sheets).  When  prompted,  say 
No  to  the  retention  of  prior  data. 

•  Copy  and  paste  data  carefully  from  the  other  data  sheets,  notably  some  or  all  of  Perspec¬ 
tives,  Cost  Data,  Selected  Details,  MRM  Level  1  Data,  Level  2  Data,  and  Level  3  Data.  Be 
sure  to  paste  into  precisely  the  correct  cells. 

•  Go  to  Summary  and  choose  Update  Summary  from  the  Options  menu. 

•  Check  the  new  Summary  sheet.  If  you  run  into  trouble,  it  may  be  easier  to  start  over  than 
to  find  the  errors  and  move  blocks  of  data  around  until  everything  works.  The  usual  prob¬ 
lems  result  from  pasting  into  the  wrong  location. 

In  other  instances,  a  user  may  want  to  copy  data  from  one  workbook  to  another  after  having 
set  up  a  new  structure  in  Template  Builder. 

The  procedure  for  setting  up  a  new  workbook  with  a  partially  new  structure,  carrying 
over  as  much  data  as  possible,  is  as  follows: 

•  Set  up  the  new  Template  Builder  in  the  fresh  version  of  PAT.  As  usual,  be  careful  in  doing 
so.  Run  Template  Builder  as  above. 

•  When  copying  and  pasting  data  from  the  previous  workbook,  recognize  that  the  new 
structure  and  the  old  structure  do  not  match.  Thus,  copying  and  pasting  should  be  done 
carefully  from  block  to  block  where  it  makes  sense.  Other  data  may  have  to  be  filled  in 
for  the  first  time  (e.g.,  for  a  new  option  or  a  new  measure). 

•  As  an  alternative,  the  user  can  copy  and  paste  full  data  from  a  previous  sheet  and  then 
edit  by  moving  columns  and  rows  around  until  they  are  in  the  right  place,  also  assuring 
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that  control  parameters  are  set  correctly.  Although  this  can  be  done,  it  is  more  error-prone 
than  the  deliberate  procedure  suggested  above. 

The  most  important  advice  is  probably  to  perform  any  data  transfer  slowly,  methodically, 
and  carefully,  rather  than  doing  it  quickly  and  trying  to  catch  and  correct  errors  afterward. 
And,  of  course,  have  backups. 


APPENDIX  C 


Editing  and  Neatening 


It  will  often  be  necessary  to  edit  PAT  sheets,  either  for  clarity  or  to  produce  an  output  display 
suitable  for  use  in  a  viewgraph  or  document.  Since  PAT  is  implemented  as  an  Excel  spreadsheet 
file,  all  of  Excel’s  ordinary  features  apply.  Elowever,  the  following  few  hints  may  prove  helpful: 

•  To  move  a  button  or  control  panel,  use  Control-Select  to  avoid  triggering  the  action. 

•  Use  Control-Select,  followed  by  the  Ungroup  command,  to  move  items  within  a  larger 
block  (e.g.,  one  menu  item  among  a  group  of  control  panels).1 

•  Use  Excel’s  custom  formats  for  charts  to  achieve  some  consistency.  Unfortunately,  such 
customization  affects  only  some  chart  attributes,  so  repetitive  editing  may  still  be  nec¬ 
essary  from  chart  to  chart.  Further,  all  formatting  is  lost  when  charts  are  regenerated. 
Unclick  the  Auto-Update  Chart  checkbox  to  avoid  losing  chart  formatting  when  switch¬ 
ing  between  sheets. 

•  To  annotate  charts  with  text-box  notes  (e.g.,  labeling  curves  or  specifying  assumptions), 
consider  copying  and  pasting  an  entire  set  of  notes  into  an  extra  worksheet  so  that  they 
can  be  retrieved  and  reused.  Regenerate  the  chart,  copy  and  paste  the  previous  notes,  and 
then  move  them  around  as  appropriate. 

•  Do  not  hesitate  to  use  custom  worksheets  to  do  “normal”  Excel  charting,  drawing  upon 
PAT’s  input  or  output  data  as  needed.  In  some  cases,  PAT’s  built-in  charts  are  not  what 
the  user  is  seeking,  and  it  is  counterproductive  to  try  to  work  around  them. 


1  We  modified  PAT  to  provide  compatibility  with  Excel  2007,  but  our  experience  with  Excel  2007  is  limited,  and  other  side 
effects  of  Microsoft’s  major  changes  may  occur.  In  particular,  we  suggest  that  users  avoid  moving  menus  and  perhaps  other 
spreadsheet  objects,  or  at  least  save  before  attempting  to  do  so.  And,  as  mentioned  earlier,  PAT  will  not  work  in  Excel  2008 
(Mac).  Macintosh  users  should  retain  Excel  2004  (even  if  they  otherwise  upgrade  to  Office  2008)  or  use  a  virtual  machine 
such  as  Parallels  Desktop  or  VMware  Fusion.  Both  solutions  have  been  quite  satisfactory,  in  our  experience. 
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